1. Empty 401 when Nova calls Neutron
- Component
- Keystone service-token permissions
- What you observe
- An instance build returns a Nova/Neutron credential error or a packet trace shows Nova calling Neutron and receiving an HTTP 401 with an empty body. Calling the same URL manually with a user token can still work.
- Why it happens
- Modern services send both the user token and their own service token. The controller config enforces service-token roles. If the Nova, Neutron, Glance or Placement service user lacks the service role in the service project, the second token is rejected.
Investigate safely
journalctl -u nova-api -b --no-pager
journalctl -u neutron-api -b --no-pager
Correct it: Grant the narrow service role to the affected service user in the service project, alongside its existing admin role, then restart the sending service so it mints a fresh token. The controller setup performs these grants when it completes.
Verify recovery: Repeat the original instance request and confirm the service-to-service call no longer returns 401. A browser login or a curl request with only the user token is not sufficient proof.
Avoid it next time: Do not disable service-token-role enforcement to make the error disappear. It removes the service-to-service identity check instead of fixing the missing grant.
2. A package is installed but no cloud service exists
- Component
- Role-package versus setup boundary
- What you observe
- The package transaction succeeded, but endpoints do not answer, no configuration exists, or no service is active.
- Why it happens
- Saphira OpenStack role packages deliberately install software only. The controller, compute and storage setup scripts perform configuration and service activation; the planning package is read-only tooling.
Investigate safely
apk info saphira-openstack-controller
saphira-openstack-setup-controller --check
Correct it: Run the appropriate setup procedure as root after completing its planning prerequisites. On a compute or storage node, use that role’s setup script instead. Do not install every role on one machine to compensate.
Verify recovery: The controller check must reach its endpoints; compute verification must show active local agents and a discovered hypervisor; storage must distinguish a missing local tgtadm command from the still-unproved Cinder volume lifecycle.
Avoid it next time: Treat installation and configuration as two deliberate stages. Read the role page before installing it on a host.
3. RabbitMQ consumers fail on a new 4.x controller
- Component
- RabbitMQ and oslo.messaging compatibility
- What you observe
- Agents or service consumers fail while declaring queues, with messages about transient non-exclusive queues. HTTP APIs may still appear healthy.
- Why it happens
- RabbitMQ 4 deprecates queue behaviour still used by oslo.messaging. Nova and Neutron need their message bus even when an API endpoint can answer an HTTP request.
Investigate safely
systemctl is-active rabbitmq-server
journalctl -u rabbitmq-server -b --no-pager
Correct it: Use the controller setup’s documented compatibility setting, deprecated_features.permit.transient_nonexcl_queues = true, for the affected RabbitMQ 4 deployment, then restart the relevant service path as required by the correction.
Verify recovery: RabbitMQ is active, agent RPC timeouts stop, and Neutron agents become live from the controller’s point of view.
Avoid it next time: Version workarounds belong to the version that needs them. Do not accumulate settings copied from another RabbitMQ release without evidence.
4. Neutron API works but agents time out
- Component
- Neutron RPC server
- What you observe
- Networks can appear in the dashboard, but OVS/DHCP agents log MessagingTimeout, ports never bind, or the controller has no live agents.
- Why it happens
- neutron-api serves HTTP. Agent work is consumed by neutron-rpc-server through RabbitMQ. Starting only the API leaves the control-plane web endpoint alive while the agent side has nobody to answer it.
Investigate safely
systemctl is-active neutron-api neutron-rpc-server rabbitmq-server
journalctl -u neutron-rpc-server -b --no-pager
journalctl -u neutron-openvswitch-agent -b --no-pager
Correct it: Restore and enable neutron-rpc-server on the controller, then correct any RabbitMQ error it reports. The controller setup starts this unit explicitly.
Verify recovery: All three checked services report active and the controller reports the compute host’s OVS/DHCP agents as alive.
Avoid it next time: Never treat an HTTP 200 from Neutron as proof that Neutron RPC and its agents work.
5. OVS reports empty bridge mappings
- Component
- Neutron OVS agent launch configuration
- What you observe
- Neutron reports no physical networks or PortBindingFailed even though openvswitch_agent.ini looks correct on disk.
- Why it happens
- The packaged agent unit can start without the plugin configuration file. In that case the OVS agent never reads the correct bridge mapping; a correct file is not a loaded configuration.
Investigate safely
systemctl status neutron-openvswitch-agent --no-pager
systemctl cat neutron-openvswitch-agent
ovs-vsctl show
Correct it: Use the compute setup script’s systemd drop-in, which starts neutron-openvswitch-agent with neutron.conf and openvswitch_agent.ini. Make the physical-network name and bridge agree exactly with the controller and the actual OVS bridge.
Verify recovery: The service is active, the bridge exists with the selected NIC, and the controller’s Neutron agent view shows the expected physical-network mapping.
Avoid it next time: Do not edit only the INI file and assume it took effect. Confirm the running unit command line and the agent’s reported mapping.
6. Neutron DHCP or Nova spawn helpers exit with permission errors
- Component
- rootwrap, privsep, sudoers and scrubbed daemon environment
- What you observe
- A service logs FailedToDropPrivileges, a privsep helper exits 1, dnsmasq loops without a lease, or a helper raises a bare FileNotFoundError although the command works in an administrator shell.
- Why it happens
- Nova and Neutron deliberately elevate only through filtered helpers. Bare sudo is denied by the narrow sudoers policy; daemon environments can also lack a PATH containing iproute2 tools.
Investigate safely
journalctl -u nova-compute -b --no-pager
journalctl -u neutron-dhcp-agent -b --no-pager
visudo -c -f /etc/sudoers.d/nova
visudo -c -f /etc/sudoers.d/neutron
Correct it: Restore the role script’s narrowly scoped Nova and Neutron sudoers fragments and Neutron root_helper setting. Fix the persistent sudo secure-path layer when diagnostics show a scrubbed environment; do not grant ALL commands to the service users.
Verify recovery: visudo validates both files, affected units stay active, DHCP/helper spawn errors stop, and a test guest can obtain its expected lease.
Avoid it next time: “Works when I run it as root” proves only the root shell environment. Test the daemon’s configured service path instead.
7. Every guest immediately fails with a KVM emulator error
- Component
- KVM availability and Nova libvirt type
- What you observe
- Guest builds fail with an error such as “Emulator does not support virt type kvm”, especially on a nested or virtualised lab host.
- Why it happens
- The host does not expose /dev/kvm, but Nova/libvirt was forced to use KVM. KVM cannot be enabled by a configuration setting when the device is absent.
Investigate safely
test -e /dev/kvm && echo 'KVM device present' || echo 'No KVM device'
grep -n '^virt_type' /etc/nova/nova.conf
journalctl -u nova-compute -b --no-pager
Correct it: Let saphira-openstack-setup-compute probe the host and select qemu when /dev/kvm is absent. Use KVM only on hosts that genuinely expose the device.
Verify recovery: Nova Compute remains active and a deliberately small test guest proceeds past the libvirt create step. QEMU emulation is a functional fallback, not a performance result.
Avoid it next time: Do not copy virt_type from another compute host. It is a per-host fact.
8. Libvirt starts, but guests fail at sockets or cgroups
- Component
- libvirt daemon set and virtqemud cgroup sandbox
- What you observe
- A build fails with a missing virtlogd socket, a handshake failure, or “Failed to create v2 cgroup: Read-only file system”.
- Why it happens
- Guests need more than virtqemud: virtlogd, virtnodedevd and virtsecretd are part of the required set. The observed packaged virtqemud sandbox also makes cgroups read-only until its targeted override is applied.
Investigate safely
systemctl is-active virtqemud virtlogd virtnodedevd virtsecretd
systemctl cat virtqemud
journalctl -u virtqemud -b --no-pager
Correct it: Run the compute setup script to start the full daemon set and install its checked ProtectControlGroups=no drop-in. Do not copy that override to unrelated units without a matching failure.
Verify recovery: All four units report active and a test build no longer fails during the libvirt handshake or cgroup creation.
Avoid it next time: Starting only the daemon whose name contains qemu is not enough for this role.
9. Instance is ACTIVE but has no usable network or DHCP lease
- Component
- Neutron DHCP, network/subnet configuration and OVS binding
- What you observe
- Nova marks the instance ACTIVE, but it has no expected address, cannot reach a gateway, or its application is unreachable.
- Why it happens
- ACTIVE confirms a successful Nova build request, not an end-to-end network. DHCP needs a subnet with DHCP enabled, a live DHCP agent, Neutron RPC, a bound port and a correct provider bridge.
Investigate safely
systemctl is-active neutron-dhcp-agent neutron-openvswitch-agent
journalctl -u neutron-dhcp-agent -b --no-pager
journalctl -u neutron-openvswitch-agent -b --no-pager
ovs-vsctl show
Correct it: Check in order: the network has the intended subnet/DHCP setting; neutron-rpc-server and RabbitMQ work; the DHCP and OVS agents are live; the agent reports the bridge mapping; then review the guest image’s own network configuration and security group.
Verify recovery: The guest has the expected lease/address and can reach the next hop or service expected by the design. Do not call an address displayed by the dashboard a reachability test.
Avoid it next time: Do not open every security-group port before proving DHCP and bridge binding. Firewall rules cannot repair a missing lease.
10. Horizon login page is unstyled or interactions fail
- Component
- Horizon static assets
- What you observe
- The dashboard loads as plain markup, styling is absent, or normal page interactions return missing static resources.
- Why it happens
- Horizon needs Django static assets collected into its configured STATIC_ROOT. The controller setup runs collectstatic after preparing the Horizon session and static directories.
Investigate safely
systemctl is-active horizon
journalctl -u horizon -b --no-pager
ls -ld /var/lib/horizon/static /var/lib/horizon/sessions
Correct it: Inspect the Horizon setup phase and ownership first. Re-run the controller setup after correcting the specific collectstatic or service problem; it contains the exact Django invocation and preserves configuration unless forced.
Verify recovery: Horizon is active, its static directory is populated and the login/dashboard render normally in a browser.
Avoid it next time: Do not diagnose the browser page as an API or JavaScript problem before checking the static-asset build step.
11. Console URL spins forever or points at a LAN address
- Component
- noVNC assets, proxy, TLS hostname and compute Nova configuration
- What you observe
- Horizon shows an instance but Console cannot load, points to http://an-internal-address:6080, or returns missing noVNC content.
- Why it happens
- The console is a chain: guest VNC on compute → Nova noVNC proxy → noVNC web assets → controller TLS reverse proxy → browser. Nova Compute mints the browser URL, so correcting only the controller cannot repair a stale compute base URL.
Investigate safely
test -f /usr/share/novnc/vnc_lite.html && echo 'noVNC assets present' || echo 'noVNC assets missing'
systemctl is-active nova-novncproxy
journalctl -u nova-novncproxy -b --no-pager
Correct it: At present, stop at the missing-assets gate rather than installing an unreviewed static binary. When the reviewed package path exists, configure noVNC assets, the proxy, TLS /vnc/ forwarding and the public HTTPS base URL on every compute host, then restart Nova Compute and request a new console URL.
Verify recovery: Only after the asset, proxy and TLS path are complete should a freshly minted HTTPS console URL load through the public hostname.
Avoid it next time: Do not publish raw port 6080 as a substitute for the intended TLS-proxied console path.
12. Storage preparation appears complete but volumes cannot attach
- Component
- Cinder LVM and iSCSI target integration
- What you observe
- The storage script created a volume group and cinder.conf, but reports STOPPING HERE, HONESTLY because tgtadm is absent; or cinder-volume starts but no volume has yet been attached and read back.
- Why it happens
- LVM allocates logical volumes, but it does not export them to compute hosts. The reviewed target chain—tgt, open-iscsi, open-isns and libiscsi—is now published, but each storage host still needs it installed and the controller still needs the Cinder database/grants. Neither package availability nor an active daemon proves the attach path.
Investigate safely
saphira-openstack-setup-storage --check
which tgtadm || true
vgs cinder-volumes
Correct it: Do not invent an export path. Retain the verified LVM state, install the published reviewed target chain as part of a controlled deployment, complete the Cinder database/grants, rerun the script, then perform a full create/attach/write/read/delete test on disposable loopback storage.
Verify recovery: The eventual proof is an actual test volume attached to an instance with data written and read back—not a successful volume-group command.
Avoid it next time: Never call a volume service production ready because LVM exists. The export and attach path is the service.
13. Scheduler says No valid host after joining compute
- Component
- Nova cell discovery and resource reporting
- What you observe
- The compute services are active locally but an instance request reports No valid host.
- Why it happens
- Nova does not schedule onto a newly reporting host until it is discovered into the cell mapping. A service process existing on a remote machine is not enough for the scheduler to select it.
Investigate safely
nova-manage cell_v2 discover_hosts --verbose
journalctl -u nova-compute -b --no-pager
Correct it: Run discovery on the controller after the compute join succeeds, then verify the administrator view reports the hypervisor up before changing flavors or adding a second compute host.
Verify recovery: The discovery command identifies the host or reports that no discovery is needed, and the hypervisor appears up in the controller’s view.
Avoid it next time: Do not compensate by creating more hosts or larger flavors before proving one discovered compute path.
14. Bridging the provider NIC disconnects the administrator
- Component
- Open vSwitch physical network change
- What you observe
- SSH becomes unreliable or disappears while the compute setup adds the selected NIC to the OVS bridge.
- Why it happens
- The selected NIC is moved under the provider bridge. If it carried the default route, the management path changes during the operation.
Investigate safely
ip route show default
ip -o link show
ovs-vsctl show
Correct it: Use a dedicated provider NIC where possible. If the only reachable path must be bridged, arrange console access and maintenance time first; the script asks for explicit YES before proceeding with the default-route interface.
Verify recovery: Management connectivity is restored through the intended interface/bridge and the OVS mapping is visible to Neutron.
Avoid it next time: A remote SSH session is not a safe substitute for an out-of-band recovery path during a network topology change.
