How do I debug OpenStack?
I stumbled upon an error that I cannot decipher: where do I find OpenStack's logs and how do I debug the system?
First time here? Check out the FAQ!
I stumbled upon an error that I cannot decipher: where do I find OpenStack's logs and how do I debug the system?
Hi,
The OpenStack Operations guide covers this:
answered 2013-04-05 19:27:03 -0600
This post is a wiki. Anyone with karma >75 is welcome to improve it.
Debugging a piece of code is an activity as important as writing it. The importance of debugging is much more noticeable on large projects like OpenStack, which require both developers and common users contribute and report when something is not going well.
Here are some pointers to collect useful logs and debugging processes.
Logging
Looking through the logs can be tricky... not if you know how to use regular expressions!
The first thing to do when we see a strange behavior that is repeated with the same input data, is to look at logs.
Logs are files, usually plain text and arranged chronologically, that contain information about the code that is executed to perform an action.
Analyzing the information found there, you may be able to detect what the problem is or, at least, where the error occurs.
The location of the logs in the system may vary depending on the implementation, but in GNU/Linux they usually are stored in /var/log.
In OpenStack the location of these logs varies according how have you deployed it: using DevStack or by hand, whether with the distribution repository packages you are using or from source.
If you're using DevStack keep in mind that by default, DevStack doesn’t log. To enable this option you will need to add the following lines to the localrc file.
LOGFILE=$DEST/stack.sh.log
– Name and location of the stack.sh script logs. Without this line the output of this script will appear in the terminal where you run it, without being stored in any location.
SCREEN_LOGDIR=$DEST/logs/screen
– Location of the screen logs. DevStack runs OpenStack services under GNU Screen, thus providing the facility to view its current execution status. The logs simply store the output of these screens..
LOGDAYS=1
– Number of days to be logged. The old logs are replaced with the latest runs.
$DEST is the location in which the logs will be located. You can set this value as you wish.
Screen logs are just a record of the screens initiated by DevStack but you can check them anytime during the execution. The information provided is very useful to debug any OpenStack service.
GNU Screen provides the same functions as the DEC VT100 terminal, plus other aggregates. If you have no experience with this terminal, make sure to write down the following shortcuts.
Attach to a screen screen -r stack
Switch screens ctrl+a <screen no.> (e.g. ctrl+a 1)
Scroll in a screen ctrl+a esc and up/down arrows to scroll
List screens ctrl+a " and select the screen with up/down arrows
Next screen ctrl+a n
Detach from the screen ctrl+a d
Manual deployment
Here you will find most services logs in /var/log/service, where service is the name of the desired service. For example, in /var/log/keystone you will find Keystone logs.
In my experience, the only service that differs from this default location is Horizon. You can find ... (more)
HI, are there any guide on debugging with eclipse and pydev with the latest branch. I have tried commented out eventlet.monkeypatch(os=False) and replaced it with eventlet.monkeypatch(all=False,socket=True,time=True,os=False) and added import sys;sys.path.append('path') but breakpoints are ignored
Here is the link to debug openstack using eclipse
http://ilearnstack.com/2013/05/30/debug-openstack-code-local-remote-with-eclipse-and-pydev-plug-in/
Just use pdb doesn't work fine with some projects like nova and or tempest, I'm not really sure, but I think it has something to do to how testr perform the tests run. Although there is valuable tip on testr page on the wiki https://wiki.openstack.org/wiki/Testr#Debugging_.28pdb.29_Tests
TL;DR: python -m testtools.run discover my.tests.regex
Please refer Grizzly docs http://docs.openstack.org/grizzly/openstack-ops/content/logging_monitoring.html
Compute specific docs here http://docs.openstack.org/grizzly/openstack-compute/admin/content/configuring-logging.html
You should be seeing logs under: /var/log/* dir under respective service.
Networking logs can be found at /var/log/quantum/
Check this guide. It seems good.
Asked: 2013-04-05 19:10:34 -0600
Seen: 6,144 times
Last updated: Jul 16 '15
Recommended hardware to learn?
How Many Physical Servers and with what Resources?
How can I connect to my VM from outside?
Why do I need to provide contact information when signing the CLA?
What should I put into Schedule A inside CCLA?
Where can I find the OpenStack Roadmap?
nova-novncproxy service not installed