test/py: support running sandbox under gdbserver

Implement command--line option --gdbserver COMM, which does two things:

a) Run the sandbox process under gdbserver, using COMM as gdbserver's
   communication channel.

b) Disables all timeouts, so that if U-Boot is halted under the debugger,
   tests don't fail. If the user gives up in the middle of a debugging
   session, they can simply CTRL-C the test script to abort it.

This allows easy debugging of test failures without having to manually
re-create the failure conditions. Usage is:

Window 1:
./test/py/test.py --bd sandbox --gdbserver localhost:1234

Window 2:
gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'

When using this option, it likely makes sense to use pytest's -k option
to limit the set of tests that are executed.

Simply running U-Boot directly under gdb (rather than gdbserver) was
also considered. However, this was rejected because:

a) gdb's output would then be processed by the test script, and likely
   confuse it causing false failures.

b) pytest by default hides stdout from tests, which would prevent the
   user from interacting with gdb.

   While gdb can be told to redirect the debugee's stdio to a separate
   PTY, this would appear to leave gdb's stdio directed at the test
   scripts and the debugee's stdio directed elsewhere, which is the
   opposite of the desired effect. Perhaps some complicated PTY muxing
   and process hierarchy could invert this. However, the current scheme
   is simple to implement and use, so it doesn't seem worth complicating
   matters.

c) Using gdbserver allows arbitrary debuggers to be used, even those with
   a GUI. If the test scripts invoked the debugger themselves, they'd have
   to know how to execute arbitary applications. While the user could hide
   this all in a wrapper script, this feels like extra complication.

An interesting future idea might be a --gdb-screen option, which could
spawn both U-Boot and gdb separately, and spawn the screen into a newly
created window under screen. Similar options could be envisaged for
creating a new xterm/... too.

--gdbserver  currently only supports sandbox, and not real hardware.
That's primarily because the test hooks are responsible for all aspects of
hardware control, so there's nothing for the test scripts themselves can
do to enable gdbserver on real hardware. We might consider introducing a
separate --disable-timeouts option to support use of debuggers on real
hardware, and having --gdbserver imply that option.

Signed-off-by: Stephen Warren <swarren@nvidia.com>

diff --git a/test/py/conftest.py b/test/py/conftest.py
index 3e162cafcc4ad7d3f3f4f8e89dc5683428b12601..09638e64a3e3c60f66f74c734460db341f12a334 100644 (file)
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -71,6 +71,9 @@ def pytest_addoption(parser):
         help='U-Boot board identity/instance')
     parser.addoption('--build', default=False, action='store_true',
         help='Compile U-Boot before running tests')
+    parser.addoption('--gdbserver', default=None,
+        help='Run sandbox under gdbserver. The argument is the channel '+
+        'over which gdbserver should communicate, e.g. localhost:1234')
 
 def pytest_configure(config):
     """pytest hook: Perform custom initialization at startup time.
@@ -110,6 +113,10 @@ def pytest_configure(config):
         persistent_data_dir = build_dir + '/persistent-data'
     mkdir_p(persistent_data_dir)
 
+    gdbserver = config.getoption('gdbserver')
+    if gdbserver and board_type != 'sandbox':
+        raise Exception('--gdbserver only supported with sandbox')
+
     import multiplexed_log
     log = multiplexed_log.Logfile(result_dir + '/test-log.html')
 
@@ -169,6 +176,7 @@ def pytest_configure(config):
     ubconfig.persistent_data_dir = persistent_data_dir
     ubconfig.board_type = board_type
     ubconfig.board_identity = board_identity
+    ubconfig.gdbserver = gdbserver
 
     env_vars = (
         'board_type',

diff --git a/test/py/tests/test_sleep.py b/test/py/tests/test_sleep.py
index 74add891c333cfb7ba4615abe448e910b9ba4415..5c1a2623fefa77c68f2aa6758001718aae64768f 100644 (file)
--- a/test/py/tests/test_sleep.py
+++ b/test/py/tests/test_sleep.py
@@ -15,6 +15,7 @@ def test_sleep(u_boot_console):
     u_boot_console.run_command('sleep %d' % sleep_time)
     tend = time.time()
     elapsed = tend - tstart
-    delta_to_expected = abs(elapsed - sleep_time)
-    # 0.25s margin is hopefully enough to account for any system overhead.
-    assert delta_to_expected < 0.25
+    assert elapsed >= sleep_time
+    if not u_boot_console.config.gdbserver:
+        # 0.25s margin is hopefully enough to account for any system overhead.
+        assert elapsed < (sleep_time + 0.25)

diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py
index 392f8cb885327041b317ef7a1c1ff834ffaca671..cc54273351774ffb76728cdc85adf4215095463a 100644 (file)
--- a/test/py/u_boot_console_base.py
+++ b/test/py/u_boot_console_base.py
@@ -300,7 +300,8 @@ class ConsoleBase(object):
             # text if LCD is enabled. This value may need tweaking in the
             # future, possibly per-test to be optimal. This works for 'help'
             # on board 'seaboard'.
-            self.p.timeout = 30000
+            if not self.config.gdbserver:
+                self.p.timeout = 30000
             self.p.logfile_read = self.logstream
             if self.config.buildconfig.get('CONFIG_SPL', False) == 'y':
                 m = self.p.expect([pattern_u_boot_spl_signon] + self.bad_patterns)

diff --git a/test/py/u_boot_console_sandbox.py b/test/py/u_boot_console_sandbox.py
index a7263f30b8cae1a1ee4eeb889ab4a81c1671fce6..3de0fe4a3b6df831239b9825c06dcfd789e676db 100644 (file)
--- a/test/py/u_boot_console_sandbox.py
+++ b/test/py/u_boot_console_sandbox.py
@@ -39,7 +39,10 @@ class ConsoleSandbox(ConsoleBase):
             A u_boot_spawn.Spawn object that is attached to U-Boot.
         """
 
-        cmd = [
+        cmd = []
+        if self.config.gdbserver:
+            cmd += ['gdbserver', self.config.gdbserver]
+        cmd += [
             self.config.build_dir + '/u-boot',
             '-d',
             self.config.build_dir + '/arch/sandbox/dts/test.dtb'

diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
index 0f52d3e945c70a4489e59f14c44c74c5184e35af..4b9e81af3efbdd8e2d96db58273f194906ef91e6 100644 (file)
--- a/test/py/u_boot_spawn.py
+++ b/test/py/u_boot_spawn.py
@@ -148,10 +148,14 @@ class Spawn(object):
                     self.buf = self.buf[posafter:]
                     return earliest_pi
                 tnow_s = time.time()
-                tdelta_ms = (tnow_s - tstart_s) * 1000
-                if tdelta_ms > self.timeout:
-                    raise Timeout()
-                events = self.poll.poll(self.timeout - tdelta_ms)
+                if self.timeout:
+                    tdelta_ms = (tnow_s - tstart_s) * 1000
+                    poll_maxwait = self.timeout - tdelta_ms
+                    if tdelta_ms > self.timeout:
+                        raise Timeout()
+                else:
+                    poll_maxwait = None
+                events = self.poll.poll(poll_maxwait)
                 if not events:
                     raise Timeout()
                 c = os.read(self.fd, 1024)