--- /dev/null
+# Copyright (c) 2013 The Chromium OS Authors.
+#
+# See file CREDITS for list of people who contributed to this
+# project.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation; either version 2 of
+# the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+# MA 02111-1307 USA
+#
+
+What is this?
+=============
+
+This tool handles building U-Boot to check that you have not broken it
+with your patch series. It can build each individual commit and report
+which boards fail on which commits, and which errors come up. It aims
+to make full use of multi-processor machines.
+
+A key feature of buildman is its output summary, which allows warnings,
+errors or image size increases in a particular commit or board to be
+quickly identified and the offending commit pinpointed. This can be a big
+help for anyone working with >10 patches at a time.
+
+
+Caveats
+=======
+
+Buildman is still in its infancy. It is already a very useful tool, but
+expect to find problems and send patches.
+
+Buildman can be stopped and restarted, in which case it will continue
+where it left off. This should happen cleanly and without side-effects.
+If not, it is a bug, for which a patch would be welcome.
+
+Buildman gets so tied up in its work that it can ignore the outside world.
+You may need to press Ctrl-C several times to quit it. Also it will print
+out various exceptions when stopped.
+
+
+Theory of Operation
+===================
+
+(please read this section in full twice or you will be perpetually confused)
+
+Buildman is a builder. It is not make, although it runs make. It does not
+produce any useful output on the terminal while building, except for
+progress information. All the output (errors, warnings and binaries if you
+are ask for them) is stored in output directories, which you can look at
+while the build is progressing, or when it is finished.
+
+Buildman produces a concise summary of which boards succeeded and failed.
+It shows which commit introduced which board failure using a simple
+red/green colour coding. Full error information can be requested, in which
+case it is de-duped and displayed against the commit that introduced the
+error. An example workflow is below.
+
+Buildman stores image size information and can report changes in image size
+from commit to commit. An example of this is below.
+
+Buildman starts multiple threads, and each thread builds for one board at
+a time. A thread starts at the first commit, configures the source for your
+board and builds it. Then it checks out the next commit and does an
+incremental build. Eventually the thread reaches the last commit and stops.
+If errors or warnings are found along the way, the thread will reconfigure
+after every commit, and your build will be very slow. This is because a
+file that produces just a warning would not normally be rebuilt in an
+incremental build.
+
+Buildman works in an entirely separate place from your U-Boot repository.
+It creates a separate working directory for each thread, and puts the
+output files in the working directory, organised by commit name and board
+name, in a two-level hierarchy.
+
+Buildman is invoked in your U-Boot directory, the one with the .git
+directory. It clones this repository into a copy for each thread, and the
+threads do not affect the state of your git repository. Any checkouts done
+by the thread affect only the working directory for that thread.
+
+Buildman automatically selects the correct toolchain for each board. You
+must supply suitable toolchains, but buildman takes care of selecting the
+right one.
+
+Buildman always builds a branch, and always builds the upstream commit as
+well, for comparison. It cannot build individual commits at present, unless
+(maybe) you point it at an empty branch. Put all your commits in a branch,
+set the branch's upstream to a valid value, and all will be well. Otherwise
+buildman will perform random actions. Use -n to check what the random
+actions might be.
+
+Buildman is optimised for building many commits at once, for many boards.
+On multi-core machines, Buildman is fast because it uses most of the
+available CPU power. When it gets to the end, or if you are building just
+a few commits or boards, it will be pretty slow. As a tip, if you don't
+plan to use your machine for anything else, you can use -T to increase the
+number of threads beyond the default.
+
+Buildman lets you build all boards, or a subset. Specify the subset using
+the board name, architecture name, SOC name, or anything else in the
+boards.cfg file. So 'at91' will build all AT91 boards (arm), powerpc will
+build all PowerPC boards.
+
+Buildman does not store intermediate object files. It optionally copies
+the binary output into a directory when a build is successful. Size
+information is always recorded. It needs a fair bit of disk space to work,
+typically 250MB per thread.
+
+
+Setting up
+==========
+
+1. Get the U-Boot source. You probably already have it, but if not these
+steps should get you started with a repo and some commits for testing.
+
+$ cd /path/to/u-boot
+$ git clone git://git.denx.de/u-boot.git .
+$ git checkout -b my-branch origin/master
+$ # Add some commits to the branch, reading for testing
+
+2. Create ~/.buildman to tell buildman where to find tool chains. As an
+example:
+
+# Buildman settings file
+
+[toolchain]
+root: /
+rest: /toolchains/*
+eldk: /opt/eldk-4.2
+
+[toolchain-alias]
+x86: i386
+blackfin: bfin
+sh: sh4
+nds32: nds32le
+openrisc: or32
+
+
+This selects the available toolchain paths. Add the base directory for
+each of your toolchains here. Buildman will search inside these directories
+and also in any '/usr' and '/usr/bin' subdirectories.
+
+Make sure the tags (here root: rest: and eldk:) are unique.
+
+The toolchain-alias section indicates that the i386 toolchain should be used
+to build x86 commits.
+
+
+2. Check the available toolchains
+
+Run this check to make sure that you have a toolchain for every architecture.
+
+$ ./tools/buildman/buildman --list-tool-chains
+Scanning for tool chains
+ - scanning path '/'
+ - looking in '/.'
+ - looking in '/bin'
+ - looking in '/usr/bin'
+ - found '/usr/bin/gcc'
+Tool chain test: OK
+ - found '/usr/bin/c89-gcc'
+Tool chain test: OK
+ - found '/usr/bin/c99-gcc'
+Tool chain test: OK
+ - found '/usr/bin/x86_64-linux-gnu-gcc'
+Tool chain test: OK
+ - scanning path '/toolchains/powerpc-linux'
+ - looking in '/toolchains/powerpc-linux/.'
+ - looking in '/toolchains/powerpc-linux/bin'
+ - found '/toolchains/powerpc-linux/bin/powerpc-linux-gcc'
+Tool chain test: OK
+ - looking in '/toolchains/powerpc-linux/usr/bin'
+ - scanning path '/toolchains/nds32le-linux-glibc-v1f'
+ - looking in '/toolchains/nds32le-linux-glibc-v1f/.'
+ - looking in '/toolchains/nds32le-linux-glibc-v1f/bin'
+ - found '/toolchains/nds32le-linux-glibc-v1f/bin/nds32le-linux-gcc'
+Tool chain test: OK
+ - looking in '/toolchains/nds32le-linux-glibc-v1f/usr/bin'
+ - scanning path '/toolchains/nios2'
+ - looking in '/toolchains/nios2/.'
+ - looking in '/toolchains/nios2/bin'
+ - found '/toolchains/nios2/bin/nios2-linux-gcc'
+Tool chain test: OK
+ - found '/toolchains/nios2/bin/nios2-linux-uclibc-gcc'
+Tool chain test: OK
+ - looking in '/toolchains/nios2/usr/bin'
+ - found '/toolchains/nios2/usr/bin/nios2-linux-gcc'
+Tool chain test: OK
+ - found '/toolchains/nios2/usr/bin/nios2-linux-uclibc-gcc'
+Tool chain test: OK
+ - scanning path '/toolchains/microblaze-unknown-linux-gnu'
+ - looking in '/toolchains/microblaze-unknown-linux-gnu/.'
+ - looking in '/toolchains/microblaze-unknown-linux-gnu/bin'
+ - found '/toolchains/microblaze-unknown-linux-gnu/bin/microblaze-unknown-linux-gnu-gcc'
+Tool chain test: OK
+ - found '/toolchains/microblaze-unknown-linux-gnu/bin/mb-linux-gcc'
+Tool chain test: OK
+ - looking in '/toolchains/microblaze-unknown-linux-gnu/usr/bin'
+ - scanning path '/toolchains/mips-linux'
+ - looking in '/toolchains/mips-linux/.'
+ - looking in '/toolchains/mips-linux/bin'
+ - found '/toolchains/mips-linux/bin/mips-linux-gcc'
+Tool chain test: OK
+ - looking in '/toolchains/mips-linux/usr/bin'
+ - scanning path '/toolchains/old'
+ - looking in '/toolchains/old/.'
+ - looking in '/toolchains/old/bin'
+ - looking in '/toolchains/old/usr/bin'
+ - scanning path '/toolchains/i386-linux'
+ - looking in '/toolchains/i386-linux/.'
+ - looking in '/toolchains/i386-linux/bin'
+ - found '/toolchains/i386-linux/bin/i386-linux-gcc'
+Tool chain test: OK
+ - looking in '/toolchains/i386-linux/usr/bin'
+ - scanning path '/toolchains/bfin-uclinux'
+ - looking in '/toolchains/bfin-uclinux/.'
+ - looking in '/toolchains/bfin-uclinux/bin'
+ - found '/toolchains/bfin-uclinux/bin/bfin-uclinux-gcc'
+Tool chain test: OK
+ - looking in '/toolchains/bfin-uclinux/usr/bin'
+ - scanning path '/toolchains/sparc-elf'
+ - looking in '/toolchains/sparc-elf/.'
+ - looking in '/toolchains/sparc-elf/bin'
+ - found '/toolchains/sparc-elf/bin/sparc-elf-gcc'
+Tool chain test: OK
+ - looking in '/toolchains/sparc-elf/usr/bin'
+ - scanning path '/toolchains/arm-2010q1'
+ - looking in '/toolchains/arm-2010q1/.'
+ - looking in '/toolchains/arm-2010q1/bin'
+ - found '/toolchains/arm-2010q1/bin/arm-none-linux-gnueabi-gcc'
+Tool chain test: OK
+ - looking in '/toolchains/arm-2010q1/usr/bin'
+ - scanning path '/toolchains/from'
+ - looking in '/toolchains/from/.'
+ - looking in '/toolchains/from/bin'
+ - looking in '/toolchains/from/usr/bin'
+ - scanning path '/toolchains/sh4-gentoo-linux-gnu'
+ - looking in '/toolchains/sh4-gentoo-linux-gnu/.'
+ - looking in '/toolchains/sh4-gentoo-linux-gnu/bin'
+ - found '/toolchains/sh4-gentoo-linux-gnu/bin/sh4-gentoo-linux-gnu-gcc'
+Tool chain test: OK
+ - looking in '/toolchains/sh4-gentoo-linux-gnu/usr/bin'
+ - scanning path '/toolchains/avr32-linux'
+ - looking in '/toolchains/avr32-linux/.'
+ - looking in '/toolchains/avr32-linux/bin'
+ - found '/toolchains/avr32-linux/bin/avr32-gcc'
+Tool chain test: OK
+ - looking in '/toolchains/avr32-linux/usr/bin'
+ - scanning path '/toolchains/m68k-linux'
+ - looking in '/toolchains/m68k-linux/.'
+ - looking in '/toolchains/m68k-linux/bin'
+ - found '/toolchains/m68k-linux/bin/m68k-linux-gcc'
+Tool chain test: OK
+ - looking in '/toolchains/m68k-linux/usr/bin'
+List of available toolchains (17):
+arm : /toolchains/arm-2010q1/bin/arm-none-linux-gnueabi-gcc
+avr32 : /toolchains/avr32-linux/bin/avr32-gcc
+bfin : /toolchains/bfin-uclinux/bin/bfin-uclinux-gcc
+c89 : /usr/bin/c89-gcc
+c99 : /usr/bin/c99-gcc
+i386 : /toolchains/i386-linux/bin/i386-linux-gcc
+m68k : /toolchains/m68k-linux/bin/m68k-linux-gcc
+mb : /toolchains/microblaze-unknown-linux-gnu/bin/mb-linux-gcc
+microblaze: /toolchains/microblaze-unknown-linux-gnu/bin/microblaze-unknown-linux-gnu-gcc
+mips : /toolchains/mips-linux/bin/mips-linux-gcc
+nds32le : /toolchains/nds32le-linux-glibc-v1f/bin/nds32le-linux-gcc
+nios2 : /toolchains/nios2/bin/nios2-linux-gcc
+powerpc : /toolchains/powerpc-linux/bin/powerpc-linux-gcc
+sandbox : /usr/bin/gcc
+sh4 : /toolchains/sh4-gentoo-linux-gnu/bin/sh4-gentoo-linux-gnu-gcc
+sparc : /toolchains/sparc-elf/bin/sparc-elf-gcc
+x86_64 : /usr/bin/x86_64-linux-gnu-gcc
+
+
+You can see that everything is covered, even some strange ones that won't
+be used (c88 and c99). This is a feature.
+
+
+How to run it
+=============
+
+First do a dry run using the -n flag: (replace <branch> with a real, local
+branch with a valid upstream)
+
+$ ./tools/buildman/buildman -b <branch> -n
+
+If it can't detect the upstream branch, try checking out the branch, and
+doing something like 'git branch --set-upstream <branch> upstream/master'
+or something similar.
+
+As an exmmple:
+
+Dry run, so not doing much. But I would do this:
+
+Building 18 commits for 1059 boards (4 threads, 1 job per thread)
+Build directory: ../lcd9b
+ 5bb3505 Merge branch 'master' of git://git.denx.de/u-boot-arm
+ c18f1b4 tegra: Use const for pinmux_config_pingroup/table()
+ 2f043ae tegra: Add display support to funcmux
+ e349900 tegra: fdt: Add pwm binding and node
+ 424a5f0 tegra: fdt: Add LCD definitions for Tegra
+ 0636ccf tegra: Add support for PWM
+ a994fe7 tegra: Add SOC support for display/lcd
+ fcd7350 tegra: Add LCD driver
+ 4d46e9d tegra: Add LCD support to Nvidia boards
+ 991bd48 arm: Add control over cachability of memory regions
+ 54e8019 lcd: Add CONFIG_LCD_ALIGNMENT to select frame buffer alignment
+ d92aff7 lcd: Add support for flushing LCD fb from dcache after update
+ dbd0677 tegra: Align LCD frame buffer to section boundary
+ 0cff9b8 tegra: Support control of cache settings for LCD
+ 9c56900 tegra: fdt: Add LCD definitions for Seaboard
+ 5cc29db lcd: Add CONFIG_CONSOLE_SCROLL_LINES option to speed console
+ cac5a23 tegra: Enable display/lcd support on Seaboard
+ 49ff541 wip
+
+Total boards to build for each commit: 1059
+
+This shows that it will build all 1059 boards, using 4 threads (because
+we have a 4-core CPU). Each thread will run with -j1, meaning that each
+make job will use a single CPU. The list of commits to be built helps you
+confirm that things look about right. Notice that buildman has chosen a
+'base' directory for you, immediately above your source tree.
+
+Buildman works entirely inside the base directory, here ../lcd9b,
+creating a working directory for each thread, and creating output
+directories for each commit and board.
+
+
+Suggested Workflow
+==================
+
+To run the build for real, take off the -n:
+
+$ ./tools/buildman/buildman -b <branch>
+
+Buildman will set up some working directories, and get started. After a
+minute or so it will settle down to a steady pace, with a display like this:
+
+Building 18 commits for 1059 boards (4 threads, 1 job per thread)
+ 528 36 124 /19062 1:13:30 : SIMPC8313_SP
+
+This means that it is building 19062 board/commit combinations. So far it
+has managed to succesfully build 528. Another 36 have built with warnings,
+and 124 more didn't build at all. Buildman expects to complete the process
+in an hour and 15 minutes. Use this time to buy a faster computer.
+
+
+To find out how the build went, ask for a summary with -s. You can do this
+either before the build completes (presumably in another terminal) or or
+afterwards. Let's work through an example of how this is used:
+
+$ ./tools/buildman/buildman -b lcd9b -s
+...
+01: Merge branch 'master' of git://git.denx.de/u-boot-arm
+ powerpc: + galaxy5200_LOWBOOT
+02: tegra: Use const for pinmux_config_pingroup/table()
+03: tegra: Add display support to funcmux
+04: tegra: fdt: Add pwm binding and node
+05: tegra: fdt: Add LCD definitions for Tegra
+06: tegra: Add support for PWM
+07: tegra: Add SOC support for display/lcd
+08: tegra: Add LCD driver
+09: tegra: Add LCD support to Nvidia boards
+10: arm: Add control over cachability of memory regions
+11: lcd: Add CONFIG_LCD_ALIGNMENT to select frame buffer alignment
+12: lcd: Add support for flushing LCD fb from dcache after update
+ arm: + lubbock
+13: tegra: Align LCD frame buffer to section boundary
+14: tegra: Support control of cache settings for LCD
+15: tegra: fdt: Add LCD definitions for Seaboard
+16: lcd: Add CONFIG_CONSOLE_SCROLL_LINES option to speed console
+17: tegra: Enable display/lcd support on Seaboard
+18: wip
+
+This shows which commits have succeeded and which have failed. In this case
+the build is still in progress so many boards are not built yet (use -u to
+see which ones). But still we can see a few failures. The galaxy5200_LOWBOOT
+never builds correctly. This could be a problem with our toolchain, or it
+could be a bug in the upstream. The good news is that we probably don't need
+to blame our commits. The bad news is it isn't tested on that board.
+
+Commit 12 broke lubbock. That's what the '+ lubbock' means. The failure
+is never fixed by a later commit, or you would see lubbock again, in green,
+without the +.
+
+To see the actual error:
+
+$ ./tools/buildman/buildman -b <branch> -se lubbock
+...
+12: lcd: Add support for flushing LCD fb from dcache after update
+ arm: + lubbock
++common/libcommon.o: In function `lcd_sync':
++/u-boot/lcd9b/.bm-work/00/common/lcd.c:120: undefined reference to `flush_dcache_range'
++arm-none-linux-gnueabi-ld: BFD (Sourcery G++ Lite 2010q1-202) 2.19.51.20090709 assertion fail /scratch/julian/2010q1-release-linux-lite/obj/binutils-src-2010q1-202-arm-none-linux-gnueabi-i686-pc-linux-gnu/bfd/elf32-arm.c:12572
++make: *** [/u-boot/lcd9b/.bm-work/00/build/u-boot] Error 139
+13: tegra: Align LCD frame buffer to section boundary
+14: tegra: Support control of cache settings for LCD
+15: tegra: fdt: Add LCD definitions for Seaboard
+16: lcd: Add CONFIG_CONSOLE_SCROLL_LINES option to speed console
+-/u-boot/lcd9b/.bm-work/00/common/lcd.c:120: undefined reference to `flush_dcache_range'
++/u-boot/lcd9b/.bm-work/00/common/lcd.c:125: undefined reference to `flush_dcache_range'
+17: tegra: Enable display/lcd support on Seaboard
+18: wip
+
+So the problem is in lcd.c, due to missing cache operations. This information
+should be enough to work out what that commit is doing to break these
+boards. (In this case pxa did not have cache operations defined).
+
+If you see error lines marked with - that means that the errors were fixed
+by that commit. Sometimes commits can be in the wrong order, so that a
+breakage is introduced for a few commits and fixed by later commits. This
+shows up clearly with buildman. You can then reorder the commits and try
+again.
+
+At commit 16, the error moves - you can see that the old error at line 120
+is fixed, but there is a new one at line 126. This is probably only because
+we added some code and moved the broken line futher down the file.
+
+If many boards have the same error, then -e will display the error only
+once. This makes the output as concise as possible.
+
+The full build output in this case is available in:
+
+../lcd9b/12_of_18_gd92aff7_lcd--Add-support-for/lubbock/
+
+ done: Indicates the build was done, and holds the return code from make.
+ This is 0 for a good build, typically 2 for a failure.
+
+ err: Output from stderr, if any. Errors and warnings appear here.
+
+ log: Output from stdout. Normally there isn't any since buildman runs
+ in silent mode for now.
+
+ toolchain: Shows information about the toolchain used for the build.
+
+ sizes: Shows image size information.
+
+It is possible to get the build output there also. Use the -k option for
+this. In that case you will also see some output files, like:
+
+ System.map toolchain u-boot u-boot.bin u-boot.map autoconf.mk
+ (also SPL versions u-boot-spl and u-boot-spl.bin if available)
+
+
+Checking Image Sizes
+====================
+
+A key requirement for U-Boot is that you keep code/data size to a minimum.
+Where a new feature increases this noticeably it should normally be put
+behind a CONFIG flag so that boards can leave it off and keep the image
+size more or less the same with each new release.
+
+To check the impact of your commits on image size, use -S. For example:
+
+$ ./tools/buildman/buildman -b us-x86 -sS
+Summary of 10 commits for 1066 boards (4 threads, 1 job per thread)
+01: MAKEALL: add support for per architecture toolchains
+02: x86: Add function to get top of usable ram
+ x86: (for 1/3 boards) text -272.0 rodata +41.0
+03: x86: Add basic cache operations
+04: x86: Permit bootstage and timer data to be used prior to relocation
+ x86: (for 1/3 boards) data +16.0
+05: x86: Add an __end symbol to signal the end of the U-Boot binary
+ x86: (for 1/3 boards) text +76.0
+06: x86: Rearrange the output input to remove BSS
+ x86: (for 1/3 boards) bss -2140.0
+07: x86: Support relocation of FDT on start-up
+ x86: + coreboot-x86
+08: x86: Add error checking to x86 relocation code
+09: x86: Adjust link device tree include file
+10: x86: Enable CONFIG_OF_CONTROL on coreboot
+
+
+You can see that image size only changed on x86, which is good because this
+series is not supposed to change any other board. From commit 7 onwards the
+build fails so we don't get code size numbers. The numbers are fractional
+because they are an average of all boards for that architecture. The
+intention is to allow you to quickly find image size problems introduced by
+your commits.
+
+Note that the 'text' region and 'rodata' are split out. You should add the
+two together to get the total read-only size (reported as the first column
+in the output from binutil's 'size' utility).
+
+A useful option is --step which lets you skip some commits. For example
+--step 2 will show the image sizes for only every 2nd commit (so it will
+compare the image sizes of the 1st, 3rd, 5th... commits). You can also use
+--step 0 which will compare only the first and last commits. This is useful
+for an overview of how your entire series affects code size.
+
+You can also use -d to see a detailed size breakdown for each board. This
+list is sorted in order from largest growth to largest reduction.
+
+It is possible to go a little further with the -B option (--bloat). This
+shows where U-Boot has bloted, breaking the size change down to the function
+level. Example output is below:
+
+$ ./tools/buildman/buildman -b us-mem4 -sSdB
+...
+19: Roll crc32 into hash infrastructure
+ arm: (for 10/10 boards) all -143.4 bss +1.2 data -4.8 rodata -48.2 text -91.6
+ paz00 : all +23 bss -4 rodata -29 text +56
+ u-boot: add: 1/0, grow: 3/-2 bytes: 168/-104 (64)
+ function old new delta
+ hash_command 80 160 +80
+ crc32_wd_buf - 56 +56
+ ext4fs_read_file 540 568 +28
+ insert_var_value_sub 688 692 +4
+ run_list_real 1996 1992 -4
+ do_mem_crc 168 68 -100
+ trimslice : all -9 bss +16 rodata -29 text +4
+ u-boot: add: 1/0, grow: 1/-3 bytes: 136/-124 (12)
+ function old new delta
+ hash_command 80 160 +80
+ crc32_wd_buf - 56 +56
+ ext4fs_iterate_dir 672 668 -4
+ ext4fs_read_file 568 548 -20
+ do_mem_crc 168 68 -100
+ whistler : all -9 bss +16 rodata -29 text +4
+ u-boot: add: 1/0, grow: 1/-3 bytes: 136/-124 (12)
+ function old new delta
+ hash_command 80 160 +80
+ crc32_wd_buf - 56 +56
+ ext4fs_iterate_dir 672 668 -4
+ ext4fs_read_file 568 548 -20
+ do_mem_crc 168 68 -100
+ seaboard : all -9 bss -28 rodata -29 text +48
+ u-boot: add: 1/0, grow: 3/-2 bytes: 160/-104 (56)
+ function old new delta
+ hash_command 80 160 +80
+ crc32_wd_buf - 56 +56
+ ext4fs_read_file 548 568 +20
+ run_list_real 1996 2000 +4
+ do_nandboot 760 756 -4
+ do_mem_crc 168 68 -100
+ colibri_t20_iris: all -9 rodata -29 text +20
+ u-boot: add: 1/0, grow: 2/-3 bytes: 140/-112 (28)
+ function old new delta
+ hash_command 80 160 +80
+ crc32_wd_buf - 56 +56
+ read_abs_bbt 204 208 +4
+ do_nandboot 760 756 -4
+ ext4fs_read_file 576 568 -8
+ do_mem_crc 168 68 -100
+ ventana : all -37 bss -12 rodata -29 text +4
+ u-boot: add: 1/0, grow: 1/-3 bytes: 136/-124 (12)
+ function old new delta
+ hash_command 80 160 +80
+ crc32_wd_buf - 56 +56
+ ext4fs_iterate_dir 672 668 -4
+ ext4fs_read_file 568 548 -20
+ do_mem_crc 168 68 -100
+ harmony : all -37 bss -16 rodata -29 text +8
+ u-boot: add: 1/0, grow: 2/-3 bytes: 140/-124 (16)
+ function old new delta
+ hash_command 80 160 +80
+ crc32_wd_buf - 56 +56
+ nand_write_oob_syndrome 428 432 +4
+ ext4fs_iterate_dir 672 668 -4
+ ext4fs_read_file 568 548 -20
+ do_mem_crc 168 68 -100
+ medcom-wide : all -417 bss +28 data -16 rodata -93 text -336
+ u-boot: add: 1/-1, grow: 1/-2 bytes: 88/-376 (-288)
+ function old new delta
+ crc32_wd_buf - 56 +56
+ do_fat_read_at 2872 2904 +32
+ hash_algo 16 - -16
+ do_mem_crc 168 68 -100
+ hash_command 420 160 -260
+ tec : all -449 bss -4 data -16 rodata -93 text -336
+ u-boot: add: 1/-1, grow: 1/-2 bytes: 88/-376 (-288)
+ function old new delta
+ crc32_wd_buf - 56 +56
+ do_fat_read_at 2872 2904 +32
+ hash_algo 16 - -16
+ do_mem_crc 168 68 -100
+ hash_command 420 160 -260
+ plutux : all -481 bss +16 data -16 rodata -93 text -388
+ u-boot: add: 1/-1, grow: 1/-3 bytes: 68/-408 (-340)
+ function old new delta
+ crc32_wd_buf - 56 +56
+ do_load_serial_bin 1688 1700 +12
+ hash_algo 16 - -16
+ do_fat_read_at 2904 2872 -32
+ do_mem_crc 168 68 -100
+ hash_command 420 160 -260
+ powerpc: (for 5/5 boards) all +37.4 data -3.2 rodata -41.8 text +82.4
+ MPC8610HPCD : all +55 rodata -29 text +84
+ u-boot: add: 1/0, grow: 0/-1 bytes: 176/-96 (80)
+ function old new delta
+ hash_command - 176 +176
+ do_mem_crc 184 88 -96
+ MPC8641HPCN : all +55 rodata -29 text +84
+ u-boot: add: 1/0, grow: 0/-1 bytes: 176/-96 (80)
+ function old new delta
+ hash_command - 176 +176
+ do_mem_crc 184 88 -96
+ MPC8641HPCN_36BIT: all +55 rodata -29 text +84
+ u-boot: add: 1/0, grow: 0/-1 bytes: 176/-96 (80)
+ function old new delta
+ hash_command - 176 +176
+ do_mem_crc 184 88 -96
+ sbc8641d : all +55 rodata -29 text +84
+ u-boot: add: 1/0, grow: 0/-1 bytes: 176/-96 (80)
+ function old new delta
+ hash_command - 176 +176
+ do_mem_crc 184 88 -96
+ xpedite517x : all -33 data -16 rodata -93 text +76
+ u-boot: add: 1/-1, grow: 0/-1 bytes: 176/-112 (64)
+ function old new delta
+ hash_command - 176 +176
+ hash_algo 16 - -16
+ do_mem_crc 184 88 -96
+...
+
+
+This shows that commit 19 has increased text size for arm (although only one
+board was built) and by 96 bytes for powerpc. This increase was offset in both
+cases by reductions in rodata and data/bss.
+
+Shown below the summary lines is the sizes for each board. Below each board
+is the sizes for each function. This information starts with:
+
+ add - number of functions added / removed
+ grow - number of functions which grew / shrunk
+ bytes - number of bytes of code added to / removed from all functions,
+ plus the total byte change in brackets
+
+The change seems to be that hash_command() has increased by more than the
+do_mem_crc() function has decreased. The function sizes typically add up to
+roughly the text area size, but note that every read-only section except
+rodata is included in 'text', so the function total does not exactly
+correspond.
+
+It is common when refactoring code for the rodata to decrease as the text size
+increases, and vice versa.
+
+
+Other options
+=============
+
+Buildman has various other command line options. Try --help to see them.
+
+
+TODO
+====
+
+This has mostly be written in my spare time as a response to my difficulties
+in testing large series of patches. Apart from tidying up there is quite a
+bit of scope for improvement. Things like better error diffs, easier access
+to log files, error display while building. Also it would be nice it buildman
+could 'hunt' for problems, perhaps by building a few boards for each arch,
+or checking commits for changed files and building only boards which use
+those files.
+
+
+Credits
+=======
+
+Thanks to Grant Grundler <grundler@chromium.org> for his ideas for improving
+the build speed by building all commits for a board instead of the other
+way around.
+
+
+
+Simon Glass
+sjg@chromium.org
+Halloween 2012
+Updated 12-12-12
+Updated 23-02-13
--- /dev/null
+# Copyright (c) 2012 The Chromium OS Authors.
+#
+# See file CREDITS for list of people who contributed to this
+# project.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation; either version 2 of
+# the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+# MA 02111-1307 USA
+#
+
+class Board:
+ """A particular board that we can build"""
+ def __init__(self, target, arch, cpu, board_name, vendor, soc, options):
+ """Create a new board type.
+
+ Args:
+ target: Target name (use make <target>_config to configure)
+ arch: Architecture name (e.g. arm)
+ cpu: Cpu name (e.g. arm1136)
+ board_name: Name of board (e.g. integrator)
+ vendor: Name of vendor (e.g. armltd)
+ soc: Name of SOC, or '' if none (e.g. mx31)
+ options: board-specific options (e.g. integratorcp:CM1136)
+ """
+ self.target = target
+ self.arch = arch
+ self.cpu = cpu
+ self.board_name = board_name
+ self.vendor = vendor
+ self.soc = soc
+ self.props = [self.target, self.arch, self.cpu, self.board_name,
+ self.vendor, self.soc]
+ self.options = options
+ self.build_it = False
+
+
+class Boards:
+ """Manage a list of boards."""
+ def __init__(self):
+ # Use a simple list here, sinc OrderedDict requires Python 2.7
+ self._boards = []
+
+ def AddBoard(self, board):
+ """Add a new board to the list.
+
+ The board's target member must not already exist in the board list.
+
+ Args:
+ board: board to add
+ """
+ self._boards.append(board)
+
+ def ReadBoards(self, fname):
+ """Read a list of boards from a board file.
+
+ Create a board object for each and add it to our _boards list.
+
+ Args:
+ fname: Filename of boards.cfg file
+ """
+ with open(fname, 'r') as fd:
+ for line in fd:
+ if line[0] == '#':
+ continue
+ fields = line.split()
+ if not fields:
+ continue
+ for upto in range(len(fields)):
+ if fields[upto] == '-':
+ fields[upto] = ''
+ while len(fields) < 7:
+ fields.append('')
+
+ board = Board(*fields)
+ self.AddBoard(board)
+
+
+ def GetList(self):
+ """Return a list of available boards.
+
+ Returns:
+ List of Board objects
+ """
+ return self._boards
+
+ def GetDict(self):
+ """Build a dictionary containing all the boards.
+
+ Returns:
+ Dictionary:
+ key is board.target
+ value is board
+ """
+ board_dict = {}
+ for board in self._boards:
+ board_dict[board.target] = board
+ return board_dict
+
+ def GetSelectedDict(self):
+ """Return a dictionary containing the selected boards
+
+ Returns:
+ List of Board objects that are marked selected
+ """
+ board_dict = {}
+ for board in self._boards:
+ if board.build_it:
+ board_dict[board.target] = board
+ return board_dict
+
+ def GetSelected(self):
+ """Return a list of selected boards
+
+ Returns:
+ List of Board objects that are marked selected
+ """
+ return [board for board in self._boards if board.build_it]
+
+ def GetSelectedNames(self):
+ """Return a list of selected boards
+
+ Returns:
+ List of board names that are marked selected
+ """
+ return [board.target for board in self._boards if board.build_it]
+
+ def SelectBoards(self, args):
+ """Mark boards selected based on args
+
+ Args:
+ List of strings specifying boards to include, either named, or
+ by their target, architecture, cpu, vendor or soc. If empty, all
+ boards are selected.
+
+ Returns:
+ Dictionary which holds the number of boards which were selected
+ due to each argument, arranged by argument.
+ """
+ result = {}
+ for arg in args:
+ result[arg] = 0
+ result['all'] = 0
+
+ for board in self._boards:
+ if args:
+ for arg in args:
+ if arg in board.props:
+ if not board.build_it:
+ board.build_it = True
+ result[arg] += 1
+ result['all'] += 1
+ else:
+ board.build_it = True
+ result['all'] += 1
+
+ return result
--- /dev/null
+# Copyright (c) 2012 The Chromium OS Authors.
+#
+# See file CREDITS for list of people who contributed to this
+# project.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation; either version 2 of
+# the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+# MA 02111-1307 USA
+#
+
+import ConfigParser
+import os
+
+
+def Setup(fname=''):
+ """Set up the buildman settings module by reading config files
+
+ Args:
+ config_fname: Config filename to read ('' for default)
+ """
+ global settings
+ global config_fname
+
+ settings = ConfigParser.SafeConfigParser()
+ config_fname = fname
+ if config_fname == '':
+ config_fname = '%s/.buildman' % os.getenv('HOME')
+ if config_fname:
+ settings.read(config_fname)
+
+def GetItems(section):
+ """Get the items from a section of the config.
+
+ Args:
+ section: name of section to retrieve
+
+ Returns:
+ List of (name, value) tuples for the section
+ """
+ try:
+ return settings.items(section)
+ except ConfigParser.NoSectionError as e:
+ print e
+ print ("Warning: No tool chains - please add a [toolchain] section "
+ "to your buildman config file %s. See README for details" %
+ config_fname)
+ return []
+ except:
+ raise
--- /dev/null
+# Copyright (c) 2013 The Chromium OS Authors.
+#
+# Bloat-o-meter code used here Copyright 2004 Matt Mackall <mpm@selenic.com>
+#
+# See file CREDITS for list of people who contributed to this
+# project.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation; either version 2 of
+# the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+# MA 02111-1307 USA
+#
+
+import collections
+import errno
+from datetime import datetime, timedelta
+import glob
+import os
+import re
+import Queue
+import shutil
+import string
+import sys
+import threading
+import time
+
+import command
+import gitutil
+import terminal
+import toolchain
+
+
+"""
+Theory of Operation
+
+Please see README for user documentation, and you should be familiar with
+that before trying to make sense of this.
+
+Buildman works by keeping the machine as busy as possible, building different
+commits for different boards on multiple CPUs at once.
+
+The source repo (self.git_dir) contains all the commits to be built. Each
+thread works on a single board at a time. It checks out the first commit,
+configures it for that board, then builds it. Then it checks out the next
+commit and builds it (typically without re-configuring). When it runs out
+of commits, it gets another job from the builder and starts again with that
+board.
+
+Clearly the builder threads could work either way - they could check out a
+commit and then built it for all boards. Using separate directories for each
+commit/board pair they could leave their build product around afterwards
+also.
+
+The intent behind building a single board for multiple commits, is to make
+use of incremental builds. Since each commit is built incrementally from
+the previous one, builds are faster. Reconfiguring for a different board
+removes all intermediate object files.
+
+Many threads can be working at once, but each has its own working directory.
+When a thread finishes a build, it puts the output files into a result
+directory.
+
+The base directory used by buildman is normally '../<branch>', i.e.
+a directory higher than the source repository and named after the branch
+being built.
+
+Within the base directory, we have one subdirectory for each commit. Within
+that is one subdirectory for each board. Within that is the build output for
+that commit/board combination.
+
+Buildman also create working directories for each thread, in a .bm-work/
+subdirectory in the base dir.
+
+As an example, say we are building branch 'us-net' for boards 'sandbox' and
+'seaboard', and say that us-net has two commits. We will have directories
+like this:
+
+us-net/ base directory
+ 01_of_02_g4ed4ebc_net--Add-tftp-speed-/
+ sandbox/
+ u-boot.bin
+ seaboard/
+ u-boot.bin
+ 02_of_02_g4ed4ebc_net--Check-tftp-comp/
+ sandbox/
+ u-boot.bin
+ seaboard/
+ u-boot.bin
+ .bm-work/
+ 00/ working directory for thread 0 (contains source checkout)
+ build/ build output
+ 01/ working directory for thread 1
+ build/ build output
+ ...
+u-boot/ source directory
+ .git/ repository
+"""
+
+# Possible build outcomes
+OUTCOME_OK, OUTCOME_WARNING, OUTCOME_ERROR, OUTCOME_UNKNOWN = range(4)
+
+# Translate a commit subject into a valid filename
+trans_valid_chars = string.maketrans("/: ", "---")
+
+
+def Mkdir(dirname):
+ """Make a directory if it doesn't already exist.
+
+ Args:
+ dirname: Directory to create
+ """
+ try:
+ os.mkdir(dirname)
+ except OSError as err:
+ if err.errno == errno.EEXIST:
+ pass
+ else:
+ raise
+
+class BuilderJob:
+ """Holds information about a job to be performed by a thread
+
+ Members:
+ board: Board object to build
+ commits: List of commit options to build.
+ """
+ def __init__(self):
+ self.board = None
+ self.commits = []
+
+
+class ResultThread(threading.Thread):
+ """This thread processes results from builder threads.
+
+ It simply passes the results on to the builder. There is only one
+ result thread, and this helps to serialise the build output.
+ """
+ def __init__(self, builder):
+ """Set up a new result thread
+
+ Args:
+ builder: Builder which will be sent each result
+ """
+ threading.Thread.__init__(self)
+ self.builder = builder
+
+ def run(self):
+ """Called to start up the result thread.
+
+ We collect the next result job and pass it on to the build.
+ """
+ while True:
+ result = self.builder.out_queue.get()
+ self.builder.ProcessResult(result)
+ self.builder.out_queue.task_done()
+
+
+class BuilderThread(threading.Thread):
+ """This thread builds U-Boot for a particular board.
+
+ An input queue provides each new job. We run 'make' to build U-Boot
+ and then pass the results on to the output queue.
+
+ Members:
+ builder: The builder which contains information we might need
+ thread_num: Our thread number (0-n-1), used to decide on a
+ temporary directory
+ """
+ def __init__(self, builder, thread_num):
+ """Set up a new builder thread"""
+ threading.Thread.__init__(self)
+ self.builder = builder
+ self.thread_num = thread_num
+
+ def Make(self, commit, brd, stage, cwd, *args, **kwargs):
+ """Run 'make' on a particular commit and board.
+
+ The source code will already be checked out, so the 'commit'
+ argument is only for information.
+
+ Args:
+ commit: Commit object that is being built
+ brd: Board object that is being built
+ stage: Stage of the build. Valid stages are:
+ distclean - can be called to clean source
+ config - called to configure for a board
+ build - the main make invocation - it does the build
+ args: A list of arguments to pass to 'make'
+ kwargs: A list of keyword arguments to pass to command.RunPipe()
+
+ Returns:
+ CommandResult object
+ """
+ return self.builder.do_make(commit, brd, stage, cwd, *args,
+ **kwargs)
+
+ def RunCommit(self, commit_upto, brd, work_dir, do_config, force_build):
+ """Build a particular commit.
+
+ If the build is already done, and we are not forcing a build, we skip
+ the build and just return the previously-saved results.
+
+ Args:
+ commit_upto: Commit number to build (0...n-1)
+ brd: Board object to build
+ work_dir: Directory to which the source will be checked out
+ do_config: True to run a make <board>_config on the source
+ force_build: Force a build even if one was previously done
+
+ Returns:
+ tuple containing:
+ - CommandResult object containing the results of the build
+ - boolean indicating whether 'make config' is still needed
+ """
+ # Create a default result - it will be overwritte by the call to
+ # self.Make() below, in the event that we do a build.
+ result = command.CommandResult()
+ result.return_code = 0
+ out_dir = os.path.join(work_dir, 'build')
+
+ # Check if the job was already completed last time
+ done_file = self.builder.GetDoneFile(commit_upto, brd.target)
+ result.already_done = os.path.exists(done_file)
+ if result.already_done and not force_build:
+ # Get the return code from that build and use it
+ with open(done_file, 'r') as fd:
+ result.return_code = int(fd.readline())
+ err_file = self.builder.GetErrFile(commit_upto, brd.target)
+ if os.path.exists(err_file) and os.stat(err_file).st_size:
+ result.stderr = 'bad'
+ else:
+ # We are going to have to build it. First, get a toolchain
+ if not self.toolchain:
+ try:
+ self.toolchain = self.builder.toolchains.Select(brd.arch)
+ except ValueError as err:
+ result.return_code = 10
+ result.stdout = ''
+ result.stderr = str(err)
+ # TODO(sjg@chromium.org): This gets swallowed, but needs
+ # to be reported.
+
+ if self.toolchain:
+ # Checkout the right commit
+ if commit_upto is not None:
+ commit = self.builder.commits[commit_upto]
+ if self.builder.checkout:
+ git_dir = os.path.join(work_dir, '.git')
+ gitutil.Checkout(commit.hash, git_dir, work_dir,
+ force=True)
+ else:
+ commit = self.builder.commit # Ick, fix this for BuildCommits()
+
+ # Set up the environment and command line
+ env = self.toolchain.MakeEnvironment()
+ Mkdir(out_dir)
+ args = ['O=build', '-s']
+ if self.builder.num_jobs is not None:
+ args.extend(['-j', str(self.builder.num_jobs)])
+ config_args = ['%s_config' % brd.target]
+ config_out = ''
+
+ # If we need to reconfigure, do that now
+ if do_config:
+ result = self.Make(commit, brd, 'distclean', work_dir,
+ 'distclean', *args, env=env)
+ result = self.Make(commit, brd, 'config', work_dir,
+ *(args + config_args), env=env)
+ config_out = result.combined
+ do_config = False # No need to configure next time
+ if result.return_code == 0:
+ result = self.Make(commit, brd, 'build', work_dir, *args,
+ env=env)
+ result.stdout = config_out + result.stdout
+ else:
+ result.return_code = 1
+ result.stderr = 'No tool chain for %s\n' % brd.arch
+ result.already_done = False
+
+ result.toolchain = self.toolchain
+ result.brd = brd
+ result.commit_upto = commit_upto
+ result.out_dir = out_dir
+ return result, do_config
+
+ def _WriteResult(self, result, keep_outputs):
+ """Write a built result to the output directory.
+
+ Args:
+ result: CommandResult object containing result to write
+ keep_outputs: True to store the output binaries, False
+ to delete them
+ """
+ # Fatal error
+ if result.return_code < 0:
+ return
+
+ # Aborted?
+ if result.stderr and 'No child processes' in result.stderr:
+ return
+
+ if result.already_done:
+ return
+
+ # Write the output and stderr
+ output_dir = self.builder._GetOutputDir(result.commit_upto)
+ Mkdir(output_dir)
+ build_dir = self.builder.GetBuildDir(result.commit_upto,
+ result.brd.target)
+ Mkdir(build_dir)
+
+ outfile = os.path.join(build_dir, 'log')
+ with open(outfile, 'w') as fd:
+ if result.stdout:
+ fd.write(result.stdout)
+
+ errfile = self.builder.GetErrFile(result.commit_upto,
+ result.brd.target)
+ if result.stderr:
+ with open(errfile, 'w') as fd:
+ fd.write(result.stderr)
+ elif os.path.exists(errfile):
+ os.remove(errfile)
+
+ if result.toolchain:
+ # Write the build result and toolchain information.
+ done_file = self.builder.GetDoneFile(result.commit_upto,
+ result.brd.target)
+ with open(done_file, 'w') as fd:
+ fd.write('%s' % result.return_code)
+ with open(os.path.join(build_dir, 'toolchain'), 'w') as fd:
+ print >>fd, 'gcc', result.toolchain.gcc
+ print >>fd, 'path', result.toolchain.path
+ print >>fd, 'cross', result.toolchain.cross
+ print >>fd, 'arch', result.toolchain.arch
+ fd.write('%s' % result.return_code)
+
+ with open(os.path.join(build_dir, 'toolchain'), 'w') as fd:
+ print >>fd, 'gcc', result.toolchain.gcc
+ print >>fd, 'path', result.toolchain.path
+
+ # Write out the image and function size information and an objdump
+ env = result.toolchain.MakeEnvironment()
+ lines = []
+ for fname in ['u-boot', 'spl/u-boot-spl']:
+ cmd = ['%snm' % self.toolchain.cross, '--size-sort', fname]
+ nm_result = command.RunPipe([cmd], capture=True,
+ capture_stderr=True, cwd=result.out_dir,
+ raise_on_error=False, env=env)
+ if nm_result.stdout:
+ nm = self.builder.GetFuncSizesFile(result.commit_upto,
+ result.brd.target, fname)
+ with open(nm, 'w') as fd:
+ print >>fd, nm_result.stdout,
+
+ cmd = ['%sobjdump' % self.toolchain.cross, '-h', fname]
+ dump_result = command.RunPipe([cmd], capture=True,
+ capture_stderr=True, cwd=result.out_dir,
+ raise_on_error=False, env=env)
+ rodata_size = ''
+ if dump_result.stdout:
+ objdump = self.builder.GetObjdumpFile(result.commit_upto,
+ result.brd.target, fname)
+ with open(objdump, 'w') as fd:
+ print >>fd, dump_result.stdout,
+ for line in dump_result.stdout.splitlines():
+ fields = line.split()
+ if len(fields) > 5 and fields[1] == '.rodata':
+ rodata_size = fields[2]
+
+ cmd = ['%ssize' % self.toolchain.cross, fname]
+ size_result = command.RunPipe([cmd], capture=True,
+ capture_stderr=True, cwd=result.out_dir,
+ raise_on_error=False, env=env)
+ if size_result.stdout:
+ lines.append(size_result.stdout.splitlines()[1] + ' ' +
+ rodata_size)
+
+ # Write out the image sizes file. This is similar to the output
+ # of binutil's 'size' utility, but it omits the header line and
+ # adds an additional hex value at the end of each line for the
+ # rodata size
+ if len(lines):
+ sizes = self.builder.GetSizesFile(result.commit_upto,
+ result.brd.target)
+ with open(sizes, 'w') as fd:
+ print >>fd, '\n'.join(lines)
+
+ # Now write the actual build output
+ if keep_outputs:
+ patterns = ['u-boot', '*.bin', 'u-boot.dtb', '*.map',
+ 'include/autoconf.mk', 'spl/u-boot-spl',
+ 'spl/u-boot-spl.bin']
+ for pattern in patterns:
+ file_list = glob.glob(os.path.join(result.out_dir, pattern))
+ for fname in file_list:
+ shutil.copy(fname, build_dir)
+
+
+ def RunJob(self, job):
+ """Run a single job
+
+ A job consists of a building a list of commits for a particular board.
+
+ Args:
+ job: Job to build
+ """
+ brd = job.board
+ work_dir = self.builder.GetThreadDir(self.thread_num)
+ self.toolchain = None
+ if job.commits:
+ # Run 'make board_config' on the first commit
+ do_config = True
+ commit_upto = 0
+ force_build = False
+ for commit_upto in range(0, len(job.commits), job.step):
+ result, request_config = self.RunCommit(commit_upto, brd,
+ work_dir, do_config,
+ force_build or self.builder.force_build)
+ failed = result.return_code or result.stderr
+ if failed and not do_config:
+ # If our incremental build failed, try building again
+ # with a reconfig.
+ if self.builder.force_config_on_failure:
+ result, request_config = self.RunCommit(commit_upto,
+ brd, work_dir, True, True)
+ do_config = request_config
+
+ # If we built that commit, then config is done. But if we got
+ # an warning, reconfig next time to force it to build the same
+ # files that created warnings this time. Otherwise an
+ # incremental build may not build the same file, and we will
+ # think that the warning has gone away.
+ # We could avoid this by using -Werror everywhere...
+ # For errors, the problem doesn't happen, since presumably
+ # the build stopped and didn't generate output, so will retry
+ # that file next time. So we could detect warnings and deal
+ # with them specially here. For now, we just reconfigure if
+ # anything goes work.
+ # Of course this is substantially slower if there are build
+ # errors/warnings (e.g. 2-3x slower even if only 10% of builds
+ # have problems).
+ if (failed and not result.already_done and not do_config and
+ self.builder.force_config_on_failure):
+ # If this build failed, try the next one with a
+ # reconfigure.
+ # Sometimes if the board_config.h file changes it can mess
+ # with dependencies, and we get:
+ # make: *** No rule to make target `include/autoconf.mk',
+ # needed by `depend'.
+ do_config = True
+ force_build = True
+ else:
+ force_build = False
+ if self.builder.force_config_on_failure:
+ if failed:
+ do_config = True
+ result.commit_upto = commit_upto
+ if result.return_code < 0:
+ raise ValueError('Interrupt')
+
+ # We have the build results, so output the result
+ self._WriteResult(result, job.keep_outputs)
+ self.builder.out_queue.put(result)
+ else:
+ # Just build the currently checked-out build
+ result = self.RunCommit(None, True)
+ result.commit_upto = self.builder.upto
+ self.builder.out_queue.put(result)
+
+ def run(self):
+ """Our thread's run function
+
+ This thread picks a job from the queue, runs it, and then goes to the
+ next job.
+ """
+ alive = True
+ while True:
+ job = self.builder.queue.get()
+ try:
+ if self.builder.active and alive:
+ self.RunJob(job)
+ except Exception as err:
+ alive = False
+ print err
+ self.builder.queue.task_done()
+
+
+class Builder:
+ """Class for building U-Boot for a particular commit.
+
+ Public members: (many should ->private)
+ active: True if the builder is active and has not been stopped
+ already_done: Number of builds already completed
+ base_dir: Base directory to use for builder
+ checkout: True to check out source, False to skip that step.
+ This is used for testing.
+ col: terminal.Color() object
+ count: Number of commits to build
+ do_make: Method to call to invoke Make
+ fail: Number of builds that failed due to error
+ force_build: Force building even if a build already exists
+ force_config_on_failure: If a commit fails for a board, disable
+ incremental building for the next commit we build for that
+ board, so that we will see all warnings/errors again.
+ git_dir: Git directory containing source repository
+ last_line_len: Length of the last line we printed (used for erasing
+ it with new progress information)
+ num_jobs: Number of jobs to run at once (passed to make as -j)
+ num_threads: Number of builder threads to run
+ out_queue: Queue of results to process
+ re_make_err: Compiled regular expression for ignore_lines
+ queue: Queue of jobs to run
+ threads: List of active threads
+ toolchains: Toolchains object to use for building
+ upto: Current commit number we are building (0.count-1)
+ warned: Number of builds that produced at least one warning
+
+ Private members:
+ _base_board_dict: Last-summarised Dict of boards
+ _base_err_lines: Last-summarised list of errors
+ _build_period_us: Time taken for a single build (float object).
+ _complete_delay: Expected delay until completion (timedelta)
+ _next_delay_update: Next time we plan to display a progress update
+ (datatime)
+ _show_unknown: Show unknown boards (those not built) in summary
+ _timestamps: List of timestamps for the completion of the last
+ last _timestamp_count builds. Each is a datetime object.
+ _timestamp_count: Number of timestamps to keep in our list.
+ _working_dir: Base working directory containing all threads
+ """
+ class Outcome:
+ """Records a build outcome for a single make invocation
+
+ Public Members:
+ rc: Outcome value (OUTCOME_...)
+ err_lines: List of error lines or [] if none
+ sizes: Dictionary of image size information, keyed by filename
+ - Each value is itself a dictionary containing
+ values for 'text', 'data' and 'bss', being the integer
+ size in bytes of each section.
+ func_sizes: Dictionary keyed by filename - e.g. 'u-boot'. Each
+ value is itself a dictionary:
+ key: function name
+ value: Size of function in bytes
+ """
+ def __init__(self, rc, err_lines, sizes, func_sizes):
+ self.rc = rc
+ self.err_lines = err_lines
+ self.sizes = sizes
+ self.func_sizes = func_sizes
+
+ def __init__(self, toolchains, base_dir, git_dir, num_threads, num_jobs,
+ checkout=True, show_unknown=True, step=1):
+ """Create a new Builder object
+
+ Args:
+ toolchains: Toolchains object to use for building
+ base_dir: Base directory to use for builder
+ git_dir: Git directory containing source repository
+ num_threads: Number of builder threads to run
+ num_jobs: Number of jobs to run at once (passed to make as -j)
+ checkout: True to check out source, False to skip that step.
+ This is used for testing.
+ show_unknown: Show unknown boards (those not built) in summary
+ step: 1 to process every commit, n to process every nth commit
+ """
+ self.toolchains = toolchains
+ self.base_dir = base_dir
+ self._working_dir = os.path.join(base_dir, '.bm-work')
+ self.threads = []
+ self.active = True
+ self.do_make = self.Make
+ self.checkout = checkout
+ self.num_threads = num_threads
+ self.num_jobs = num_jobs
+ self.already_done = 0
+ self.force_build = False
+ self.git_dir = git_dir
+ self._show_unknown = show_unknown
+ self._timestamp_count = 10
+ self._build_period_us = None
+ self._complete_delay = None
+ self._next_delay_update = datetime.now()
+ self.force_config_on_failure = True
+ self._step = step
+
+ self.col = terminal.Color()
+
+ self.queue = Queue.Queue()
+ self.out_queue = Queue.Queue()
+ for i in range(self.num_threads):
+ t = BuilderThread(self, i)
+ t.setDaemon(True)
+ t.start()
+ self.threads.append(t)
+
+ self.last_line_len = 0
+ t = ResultThread(self)
+ t.setDaemon(True)
+ t.start()
+ self.threads.append(t)
+
+ ignore_lines = ['(make.*Waiting for unfinished)', '(Segmentation fault)']
+ self.re_make_err = re.compile('|'.join(ignore_lines))
+
+ def __del__(self):
+ """Get rid of all threads created by the builder"""
+ for t in self.threads:
+ del t
+
+ def _AddTimestamp(self):
+ """Add a new timestamp to the list and record the build period.
+
+ The build period is the length of time taken to perform a single
+ build (one board, one commit).
+ """
+ now = datetime.now()
+ self._timestamps.append(now)
+ count = len(self._timestamps)
+ delta = self._timestamps[-1] - self._timestamps[0]
+ seconds = delta.total_seconds()
+
+ # If we have enough data, estimate build period (time taken for a
+ # single build) and therefore completion time.
+ if count > 1 and self._next_delay_update < now:
+ self._next_delay_update = now + timedelta(seconds=2)
+ if seconds > 0:
+ self._build_period = float(seconds) / count
+ todo = self.count - self.upto
+ self._complete_delay = timedelta(microseconds=
+ self._build_period * todo * 1000000)
+ # Round it
+ self._complete_delay -= timedelta(
+ microseconds=self._complete_delay.microseconds)
+
+ if seconds > 60:
+ self._timestamps.popleft()
+ count -= 1
+
+ def ClearLine(self, length):
+ """Clear any characters on the current line
+
+ Make way for a new line of length 'length', by outputting enough
+ spaces to clear out the old line. Then remember the new length for
+ next time.
+
+ Args:
+ length: Length of new line, in characters
+ """
+ if length < self.last_line_len:
+ print ' ' * (self.last_line_len - length),
+ print '\r',
+ self.last_line_len = length
+ sys.stdout.flush()
+
+ def SelectCommit(self, commit, checkout=True):
+ """Checkout the selected commit for this build
+ """
+ self.commit = commit
+ if checkout and self.checkout:
+ gitutil.Checkout(commit.hash)
+
+ def Make(self, commit, brd, stage, cwd, *args, **kwargs):
+ """Run make
+
+ Args:
+ commit: Commit object that is being built
+ brd: Board object that is being built
+ stage: Stage that we are at (distclean, config, build)
+ cwd: Directory where make should be run
+ args: Arguments to pass to make
+ kwargs: Arguments to pass to command.RunPipe()
+ """
+ cmd = ['make'] + list(args)
+ result = command.RunPipe([cmd], capture=True, capture_stderr=True,
+ cwd=cwd, raise_on_error=False, **kwargs)
+ return result
+
+ def ProcessResult(self, result):
+ """Process the result of a build, showing progress information
+
+ Args:
+ result: A CommandResult object
+ """
+ col = terminal.Color()
+ if result:
+ target = result.brd.target
+
+ if result.return_code < 0:
+ self.active = False
+ command.StopAll()
+ return
+
+ self.upto += 1
+ if result.return_code != 0:
+ self.fail += 1
+ elif result.stderr:
+ self.warned += 1
+ if result.already_done:
+ self.already_done += 1
+ else:
+ target = '(starting)'
+
+ # Display separate counts for ok, warned and fail
+ ok = self.upto - self.warned - self.fail
+ line = '\r' + self.col.Color(self.col.GREEN, '%5d' % ok)
+ line += self.col.Color(self.col.YELLOW, '%5d' % self.warned)
+ line += self.col.Color(self.col.RED, '%5d' % self.fail)
+
+ name = ' /%-5d ' % self.count
+
+ # Add our current completion time estimate
+ self._AddTimestamp()
+ if self._complete_delay:
+ name += '%s : ' % self._complete_delay
+ # When building all boards for a commit, we can print a commit
+ # progress message.
+ if result and result.commit_upto is None:
+ name += 'commit %2d/%-3d' % (self.commit_upto + 1,
+ self.commit_count)
+
+ name += target
+ print line + name,
+ length = 13 + len(name)
+ self.ClearLine(length)
+
+ def _GetOutputDir(self, commit_upto):
+ """Get the name of the output directory for a commit number
+
+ The output directory is typically .../<branch>/<commit>.
+
+ Args:
+ commit_upto: Commit number to use (0..self.count-1)
+ """
+ commit = self.commits[commit_upto]
+ subject = commit.subject.translate(trans_valid_chars)
+ commit_dir = ('%02d_of_%02d_g%s_%s' % (commit_upto + 1,
+ self.commit_count, commit.hash, subject[:20]))
+ output_dir = os.path.join(self.base_dir, commit_dir)
+ return output_dir
+
+ def GetBuildDir(self, commit_upto, target):
+ """Get the name of the build directory for a commit number
+
+ The build directory is typically .../<branch>/<commit>/<target>.
+
+ Args:
+ commit_upto: Commit number to use (0..self.count-1)
+ target: Target name
+ """
+ output_dir = self._GetOutputDir(commit_upto)
+ return os.path.join(output_dir, target)
+
+ def GetDoneFile(self, commit_upto, target):
+ """Get the name of the done file for a commit number
+
+ Args:
+ commit_upto: Commit number to use (0..self.count-1)
+ target: Target name
+ """
+ return os.path.join(self.GetBuildDir(commit_upto, target), 'done')
+
+ def GetSizesFile(self, commit_upto, target):
+ """Get the name of the sizes file for a commit number
+
+ Args:
+ commit_upto: Commit number to use (0..self.count-1)
+ target: Target name
+ """
+ return os.path.join(self.GetBuildDir(commit_upto, target), 'sizes')
+
+ def GetFuncSizesFile(self, commit_upto, target, elf_fname):
+ """Get the name of the funcsizes file for a commit number and ELF file
+
+ Args:
+ commit_upto: Commit number to use (0..self.count-1)
+ target: Target name
+ elf_fname: Filename of elf image
+ """
+ return os.path.join(self.GetBuildDir(commit_upto, target),
+ '%s.sizes' % elf_fname.replace('/', '-'))
+
+ def GetObjdumpFile(self, commit_upto, target, elf_fname):
+ """Get the name of the objdump file for a commit number and ELF file
+
+ Args:
+ commit_upto: Commit number to use (0..self.count-1)
+ target: Target name
+ elf_fname: Filename of elf image
+ """
+ return os.path.join(self.GetBuildDir(commit_upto, target),
+ '%s.objdump' % elf_fname.replace('/', '-'))
+
+ def GetErrFile(self, commit_upto, target):
+ """Get the name of the err file for a commit number
+
+ Args:
+ commit_upto: Commit number to use (0..self.count-1)
+ target: Target name
+ """
+ output_dir = self.GetBuildDir(commit_upto, target)
+ return os.path.join(output_dir, 'err')
+
+ def FilterErrors(self, lines):
+ """Filter out errors in which we have no interest
+
+ We should probably use map().
+
+ Args:
+ lines: List of error lines, each a string
+ Returns:
+ New list with only interesting lines included
+ """
+ out_lines = []
+ for line in lines:
+ if not self.re_make_err.search(line):
+ out_lines.append(line)
+ return out_lines
+
+ def ReadFuncSizes(self, fname, fd):
+ """Read function sizes from the output of 'nm'
+
+ Args:
+ fd: File containing data to read
+ fname: Filename we are reading from (just for errors)
+
+ Returns:
+ Dictionary containing size of each function in bytes, indexed by
+ function name.
+ """
+ sym = {}
+ for line in fd.readlines():
+ try:
+ size, type, name = line[:-1].split()
+ except:
+ print "Invalid line in file '%s': '%s'" % (fname, line[:-1])
+ continue
+ if type in 'tTdDbB':
+ # function names begin with '.' on 64-bit powerpc
+ if '.' in name[1:]:
+ name = 'static.' + name.split('.')[0]
+ sym[name] = sym.get(name, 0) + int(size, 16)
+ return sym
+
+ def GetBuildOutcome(self, commit_upto, target, read_func_sizes):
+ """Work out the outcome of a build.
+
+ Args:
+ commit_upto: Commit number to check (0..n-1)
+ target: Target board to check
+ read_func_sizes: True to read function size information
+
+ Returns:
+ Outcome object
+ """
+ done_file = self.GetDoneFile(commit_upto, target)
+ sizes_file = self.GetSizesFile(commit_upto, target)
+ sizes = {}
+ func_sizes = {}
+ if os.path.exists(done_file):
+ with open(done_file, 'r') as fd:
+ return_code = int(fd.readline())
+ err_lines = []
+ err_file = self.GetErrFile(commit_upto, target)
+ if os.path.exists(err_file):
+ with open(err_file, 'r') as fd:
+ err_lines = self.FilterErrors(fd.readlines())
+
+ # Decide whether the build was ok, failed or created warnings
+ if return_code:
+ rc = OUTCOME_ERROR
+ elif len(err_lines):
+ rc = OUTCOME_WARNING
+ else:
+ rc = OUTCOME_OK
+
+ # Convert size information to our simple format
+ if os.path.exists(sizes_file):
+ with open(sizes_file, 'r') as fd:
+ for line in fd.readlines():
+ values = line.split()
+ rodata = 0
+ if len(values) > 6:
+ rodata = int(values[6], 16)
+ size_dict = {
+ 'all' : int(values[0]) + int(values[1]) +
+ int(values[2]),
+ 'text' : int(values[0]) - rodata,
+ 'data' : int(values[1]),
+ 'bss' : int(values[2]),
+ 'rodata' : rodata,
+ }
+ sizes[values[5]] = size_dict
+
+ if read_func_sizes:
+ pattern = self.GetFuncSizesFile(commit_upto, target, '*')
+ for fname in glob.glob(pattern):
+ with open(fname, 'r') as fd:
+ dict_name = os.path.basename(fname).replace('.sizes',
+ '')
+ func_sizes[dict_name] = self.ReadFuncSizes(fname, fd)
+
+ return Builder.Outcome(rc, err_lines, sizes, func_sizes)
+
+ return Builder.Outcome(OUTCOME_UNKNOWN, [], {}, {})
+
+ def GetResultSummary(self, boards_selected, commit_upto, read_func_sizes):
+ """Calculate a summary of the results of building a commit.
+
+ Args:
+ board_selected: Dict containing boards to summarise
+ commit_upto: Commit number to summarize (0..self.count-1)
+ read_func_sizes: True to read function size information
+
+ Returns:
+ Tuple:
+ Dict containing boards which passed building this commit.
+ keyed by board.target
+ List containing a summary of error/warning lines
+ """
+ board_dict = {}
+ err_lines_summary = []
+
+ for board in boards_selected.itervalues():
+ outcome = self.GetBuildOutcome(commit_upto, board.target,
+ read_func_sizes)
+ board_dict[board.target] = outcome
+ for err in outcome.err_lines:
+ if err and not err.rstrip() in err_lines_summary:
+ err_lines_summary.append(err.rstrip())
+ return board_dict, err_lines_summary
+
+ def AddOutcome(self, board_dict, arch_list, changes, char, color):
+ """Add an output to our list of outcomes for each architecture
+
+ This simple function adds failing boards (changes) to the
+ relevant architecture string, so we can print the results out
+ sorted by architecture.
+
+ Args:
+ board_dict: Dict containing all boards
+ arch_list: Dict keyed by arch name. Value is a string containing
+ a list of board names which failed for that arch.
+ changes: List of boards to add to arch_list
+ color: terminal.Colour object
+ """
+ done_arch = {}
+ for target in changes:
+ if target in board_dict:
+ arch = board_dict[target].arch
+ else:
+ arch = 'unknown'
+ str = self.col.Color(color, ' ' + target)
+ if not arch in done_arch:
+ str = self.col.Color(color, char) + ' ' + str
+ done_arch[arch] = True
+ if not arch in arch_list:
+ arch_list[arch] = str
+ else:
+ arch_list[arch] += str
+
+
+ def ColourNum(self, num):
+ color = self.col.RED if num > 0 else self.col.GREEN
+ if num == 0:
+ return '0'
+ return self.col.Color(color, str(num))
+
+ def ResetResultSummary(self, board_selected):
+ """Reset the results summary ready for use.
+
+ Set up the base board list to be all those selected, and set the
+ error lines to empty.
+
+ Following this, calls to PrintResultSummary() will use this
+ information to work out what has changed.
+
+ Args:
+ board_selected: Dict containing boards to summarise, keyed by
+ board.target
+ """
+ self._base_board_dict = {}
+ for board in board_selected:
+ self._base_board_dict[board] = Builder.Outcome(0, [], [], {})
+ self._base_err_lines = []
+
+ def PrintFuncSizeDetail(self, fname, old, new):
+ grow, shrink, add, remove, up, down = 0, 0, 0, 0, 0, 0
+ delta, common = [], {}
+
+ for a in old:
+ if a in new:
+ common[a] = 1
+
+ for name in old:
+ if name not in common:
+ remove += 1
+ down += old[name]
+ delta.append([-old[name], name])
+
+ for name in new:
+ if name not in common:
+ add += 1
+ up += new[name]
+ delta.append([new[name], name])
+
+ for name in common:
+ diff = new.get(name, 0) - old.get(name, 0)
+ if diff > 0:
+ grow, up = grow + 1, up + diff
+ elif diff < 0:
+ shrink, down = shrink + 1, down - diff
+ delta.append([diff, name])
+
+ delta.sort()
+ delta.reverse()
+
+ args = [add, -remove, grow, -shrink, up, -down, up - down]
+ if max(args) == 0:
+ return
+ args = [self.ColourNum(x) for x in args]
+ indent = ' ' * 15
+ print ('%s%s: add: %s/%s, grow: %s/%s bytes: %s/%s (%s)' %
+ tuple([indent, self.col.Color(self.col.YELLOW, fname)] + args))
+ print '%s %-38s %7s %7s %+7s' % (indent, 'function', 'old', 'new',
+ 'delta')
+ for diff, name in delta:
+ if diff:
+ color = self.col.RED if diff > 0 else self.col.GREEN
+ msg = '%s %-38s %7s %7s %+7d' % (indent, name,
+ old.get(name, '-'), new.get(name,'-'), diff)
+ print self.col.Color(color, msg)
+
+
+ def PrintSizeDetail(self, target_list, show_bloat):
+ """Show details size information for each board
+
+ Args:
+ target_list: List of targets, each a dict containing:
+ 'target': Target name
+ 'total_diff': Total difference in bytes across all areas
+ <part_name>: Difference for that part
+ show_bloat: Show detail for each function
+ """
+ targets_by_diff = sorted(target_list, reverse=True,
+ key=lambda x: x['_total_diff'])
+ for result in targets_by_diff:
+ printed_target = False
+ for name in sorted(result):
+ diff = result[name]
+ if name.startswith('_'):
+ continue
+ if diff != 0:
+ color = self.col.RED if diff > 0 else self.col.GREEN
+ msg = ' %s %+d' % (name, diff)
+ if not printed_target:
+ print '%10s %-15s:' % ('', result['_target']),
+ printed_target = True
+ print self.col.Color(color, msg),
+ if printed_target:
+ print
+ if show_bloat:
+ target = result['_target']
+ outcome = result['_outcome']
+ base_outcome = self._base_board_dict[target]
+ for fname in outcome.func_sizes:
+ self.PrintFuncSizeDetail(fname,
+ base_outcome.func_sizes[fname],
+ outcome.func_sizes[fname])
+
+
+ def PrintSizeSummary(self, board_selected, board_dict, show_detail,
+ show_bloat):
+ """Print a summary of image sizes broken down by section.
+
+ The summary takes the form of one line per architecture. The
+ line contains deltas for each of the sections (+ means the section
+ got bigger, - means smaller). The nunmbers are the average number
+ of bytes that a board in this section increased by.
+
+ For example:
+ powerpc: (622 boards) text -0.0
+ arm: (285 boards) text -0.0
+ nds32: (3 boards) text -8.0
+
+ Args:
+ board_selected: Dict containing boards to summarise, keyed by
+ board.target
+ board_dict: Dict containing boards for which we built this
+ commit, keyed by board.target. The value is an Outcome object.
+ show_detail: Show detail for each board
+ show_bloat: Show detail for each function
+ """
+ arch_list = {}
+ arch_count = {}
+
+ # Calculate changes in size for different image parts
+ # The previous sizes are in Board.sizes, for each board
+ for target in board_dict:
+ if target not in board_selected:
+ continue
+ base_sizes = self._base_board_dict[target].sizes
+ outcome = board_dict[target]
+ sizes = outcome.sizes
+
+ # Loop through the list of images, creating a dict of size
+ # changes for each image/part. We end up with something like
+ # {'target' : 'snapper9g45, 'data' : 5, 'u-boot-spl:text' : -4}
+ # which means that U-Boot data increased by 5 bytes and SPL
+ # text decreased by 4.
+ err = {'_target' : target}
+ for image in sizes:
+ if image in base_sizes:
+ base_image = base_sizes[image]
+ # Loop through the text, data, bss parts
+ for part in sorted(sizes[image]):
+ diff = sizes[image][part] - base_image[part]
+ col = None
+ if diff:
+ if image == 'u-boot':
+ name = part
+ else:
+ name = image + ':' + part
+ err[name] = diff
+ arch = board_selected[target].arch
+ if not arch in arch_count:
+ arch_count[arch] = 1
+ else:
+ arch_count[arch] += 1
+ if not sizes:
+ pass # Only add to our list when we have some stats
+ elif not arch in arch_list:
+ arch_list[arch] = [err]
+ else:
+ arch_list[arch].append(err)
+
+ # We now have a list of image size changes sorted by arch
+ # Print out a summary of these
+ for arch, target_list in arch_list.iteritems():
+ # Get total difference for each type
+ totals = {}
+ for result in target_list:
+ total = 0
+ for name, diff in result.iteritems():
+ if name.startswith('_'):
+ continue
+ total += diff
+ if name in totals:
+ totals[name] += diff
+ else:
+ totals[name] = diff
+ result['_total_diff'] = total
+ result['_outcome'] = board_dict[result['_target']]
+
+ count = len(target_list)
+ printed_arch = False
+ for name in sorted(totals):
+ diff = totals[name]
+ if diff:
+ # Display the average difference in this name for this
+ # architecture
+ avg_diff = float(diff) / count
+ color = self.col.RED if avg_diff > 0 else self.col.GREEN
+ msg = ' %s %+1.1f' % (name, avg_diff)
+ if not printed_arch:
+ print '%10s: (for %d/%d boards)' % (arch, count,
+ arch_count[arch]),
+ printed_arch = True
+ print self.col.Color(color, msg),
+
+ if printed_arch:
+ print
+ if show_detail:
+ self.PrintSizeDetail(target_list, show_bloat)
+
+
+ def PrintResultSummary(self, board_selected, board_dict, err_lines,
+ show_sizes, show_detail, show_bloat):
+ """Compare results with the base results and display delta.
+
+ Only boards mentioned in board_selected will be considered. This
+ function is intended to be called repeatedly with the results of
+ each commit. It therefore shows a 'diff' between what it saw in
+ the last call and what it sees now.
+
+ Args:
+ board_selected: Dict containing boards to summarise, keyed by
+ board.target
+ board_dict: Dict containing boards for which we built this
+ commit, keyed by board.target. The value is an Outcome object.
+ err_lines: A list of errors for this commit, or [] if there is
+ none, or we don't want to print errors
+ show_sizes: Show image size deltas
+ show_detail: Show detail for each board
+ show_bloat: Show detail for each function
+ """
+ better = [] # List of boards fixed since last commit
+ worse = [] # List of new broken boards since last commit
+ new = [] # List of boards that didn't exist last time
+ unknown = [] # List of boards that were not built
+
+ for target in board_dict:
+ if target not in board_selected:
+ continue
+
+ # If the board was built last time, add its outcome to a list
+ if target in self._base_board_dict:
+ base_outcome = self._base_board_dict[target].rc
+ outcome = board_dict[target]
+ if outcome.rc == OUTCOME_UNKNOWN:
+ unknown.append(target)
+ elif outcome.rc < base_outcome:
+ better.append(target)
+ elif outcome.rc > base_outcome:
+ worse.append(target)
+ else:
+ new.append(target)
+
+ # Get a list of errors that have appeared, and disappeared
+ better_err = []
+ worse_err = []
+ for line in err_lines:
+ if line not in self._base_err_lines:
+ worse_err.append('+' + line)
+ for line in self._base_err_lines:
+ if line not in err_lines:
+ better_err.append('-' + line)
+
+ # Display results by arch
+ if better or worse or unknown or new or worse_err or better_err:
+ arch_list = {}
+ self.AddOutcome(board_selected, arch_list, better, '',
+ self.col.GREEN)
+ self.AddOutcome(board_selected, arch_list, worse, '+',
+ self.col.RED)
+ self.AddOutcome(board_selected, arch_list, new, '*', self.col.BLUE)
+ if self._show_unknown:
+ self.AddOutcome(board_selected, arch_list, unknown, '?',
+ self.col.MAGENTA)
+ for arch, target_list in arch_list.iteritems():
+ print '%10s: %s' % (arch, target_list)
+ if better_err:
+ print self.col.Color(self.col.GREEN, '\n'.join(better_err))
+ if worse_err:
+ print self.col.Color(self.col.RED, '\n'.join(worse_err))
+
+ if show_sizes:
+ self.PrintSizeSummary(board_selected, board_dict, show_detail,
+ show_bloat)
+
+ # Save our updated information for the next call to this function
+ self._base_board_dict = board_dict
+ self._base_err_lines = err_lines
+
+ # Get a list of boards that did not get built, if needed
+ not_built = []
+ for board in board_selected:
+ if not board in board_dict:
+ not_built.append(board)
+ if not_built:
+ print "Boards not built (%d): %s" % (len(not_built),
+ ', '.join(not_built))
+
+
+ def ShowSummary(self, commits, board_selected, show_errors, show_sizes,
+ show_detail, show_bloat):
+ """Show a build summary for U-Boot for a given board list.
+
+ Reset the result summary, then repeatedly call GetResultSummary on
+ each commit's results, then display the differences we see.
+
+ Args:
+ commit: Commit objects to summarise
+ board_selected: Dict containing boards to summarise
+ show_errors: Show errors that occured
+ show_sizes: Show size deltas
+ show_detail: Show detail for each board
+ show_bloat: Show detail for each function
+ """
+ self.commit_count = len(commits)
+ self.commits = commits
+ self.ResetResultSummary(board_selected)
+
+ for commit_upto in range(0, self.commit_count, self._step):
+ board_dict, err_lines = self.GetResultSummary(board_selected,
+ commit_upto, read_func_sizes=show_bloat)
+ msg = '%02d: %s' % (commit_upto + 1, commits[commit_upto].subject)
+ print self.col.Color(self.col.BLUE, msg)
+ self.PrintResultSummary(board_selected, board_dict,
+ err_lines if show_errors else [], show_sizes, show_detail,
+ show_bloat)
+
+
+ def SetupBuild(self, board_selected, commits):
+ """Set up ready to start a build.
+
+ Args:
+ board_selected: Selected boards to build
+ commits: Selected commits to build
+ """
+ # First work out how many commits we will build
+ count = (len(commits) + self._step - 1) / self._step
+ self.count = len(board_selected) * count
+ self.upto = self.warned = self.fail = 0
+ self._timestamps = collections.deque()
+
+ def BuildBoardsForCommit(self, board_selected, keep_outputs):
+ """Build all boards for a single commit"""
+ self.SetupBuild(board_selected)
+ self.count = len(board_selected)
+ for brd in board_selected.itervalues():
+ job = BuilderJob()
+ job.board = brd
+ job.commits = None
+ job.keep_outputs = keep_outputs
+ self.queue.put(brd)
+
+ self.queue.join()
+ self.out_queue.join()
+ print
+ self.ClearLine(0)
+
+ def BuildCommits(self, commits, board_selected, show_errors, keep_outputs):
+ """Build all boards for all commits (non-incremental)"""
+ self.commit_count = len(commits)
+
+ self.ResetResultSummary(board_selected)
+ for self.commit_upto in range(self.commit_count):
+ self.SelectCommit(commits[self.commit_upto])
+ self.SelectOutputDir()
+ Mkdir(self.output_dir)
+
+ self.BuildBoardsForCommit(board_selected, keep_outputs)
+ board_dict, err_lines = self.GetResultSummary()
+ self.PrintResultSummary(board_selected, board_dict,
+ err_lines if show_errors else [])
+
+ if self.already_done:
+ print '%d builds already done' % self.already_done
+
+ def GetThreadDir(self, thread_num):
+ """Get the directory path to the working dir for a thread.
+
+ Args:
+ thread_num: Number of thread to check.
+ """
+ return os.path.join(self._working_dir, '%02d' % thread_num)
+
+ def _PrepareThread(self, thread_num):
+ """Prepare the working directory for a thread.
+
+ This clones or fetches the repo into the thread's work directory.
+
+ Args:
+ thread_num: Thread number (0, 1, ...)
+ """
+ thread_dir = self.GetThreadDir(thread_num)
+ Mkdir(thread_dir)
+ git_dir = os.path.join(thread_dir, '.git')
+
+ # Clone the repo if it doesn't already exist
+ # TODO(sjg@chromium): Perhaps some git hackery to symlink instead, so
+ # we have a private index but uses the origin repo's contents?
+ if self.git_dir:
+ src_dir = os.path.abspath(self.git_dir)
+ if os.path.exists(git_dir):
+ gitutil.Fetch(git_dir, thread_dir)
+ else:
+ print 'Cloning repo for thread %d' % thread_num
+ gitutil.Clone(src_dir, thread_dir)
+
+ def _PrepareWorkingSpace(self, max_threads):
+ """Prepare the working directory for use.
+
+ Set up the git repo for each thread.
+
+ Args:
+ max_threads: Maximum number of threads we expect to need.
+ """
+ Mkdir(self._working_dir)
+ for thread in range(max_threads):
+ self._PrepareThread(thread)
+
+ def _PrepareOutputSpace(self):
+ """Get the output directories ready to receive files.
+
+ We delete any output directories which look like ones we need to
+ create. Having left over directories is confusing when the user wants
+ to check the output manually.
+ """
+ dir_list = []
+ for commit_upto in range(self.commit_count):
+ dir_list.append(self._GetOutputDir(commit_upto))
+
+ for dirname in glob.glob(os.path.join(self.base_dir, '*')):
+ if dirname not in dir_list:
+ shutil.rmtree(dirname)
+
+ def BuildBoards(self, commits, board_selected, show_errors, keep_outputs):
+ """Build all commits for a list of boards
+
+ Args:
+ commits: List of commits to be build, each a Commit object
+ boards_selected: Dict of selected boards, key is target name,
+ value is Board object
+ show_errors: True to show summarised error/warning info
+ keep_outputs: True to save build output files
+ """
+ self.commit_count = len(commits)
+ self.commits = commits
+
+ self.ResetResultSummary(board_selected)
+ Mkdir(self.base_dir)
+ self._PrepareWorkingSpace(min(self.num_threads, len(board_selected)))
+ self._PrepareOutputSpace()
+ self.SetupBuild(board_selected, commits)
+ self.ProcessResult(None)
+
+ # Create jobs to build all commits for each board
+ for brd in board_selected.itervalues():
+ job = BuilderJob()
+ job.board = brd
+ job.commits = commits
+ job.keep_outputs = keep_outputs
+ job.step = self._step
+ self.queue.put(job)
+
+ # Wait until all jobs are started
+ self.queue.join()
+
+ # Wait until we have processed all output
+ self.out_queue.join()
+ print
+ self.ClearLine(0)
--- /dev/null
+buildman.py
\ No newline at end of file
--- /dev/null
+#!/usr/bin/python
+#
+# Copyright (c) 2012 The Chromium OS Authors.
+#
+# See file CREDITS for list of people who contributed to this
+# project.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation; either version 2 of
+# the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+# MA 02111-1307 USA
+#
+
+"""See README for more information"""
+
+import multiprocessing
+from optparse import OptionParser
+import os
+import re
+import sys
+import unittest
+
+# Bring in the patman libraries
+our_path = os.path.dirname(os.path.realpath(__file__))
+sys.path.append(os.path.join(our_path, '../patman'))
+
+# Our modules
+import board
+import builder
+import checkpatch
+import command
+import control
+import doctest
+import gitutil
+import patchstream
+import terminal
+import toolchain
+
+def RunTests():
+ import test
+
+ sys.argv = [sys.argv[0]]
+ suite = unittest.TestLoader().loadTestsFromTestCase(test.TestBuild)
+ result = unittest.TestResult()
+ suite.run(result)
+
+ # TODO: Surely we can just 'print' result?
+ print result
+ for test, err in result.errors:
+ print err
+ for test, err in result.failures:
+ print err
+
+
+parser = OptionParser()
+parser.add_option('-b', '--branch', type='string',
+ help='Branch name to build')
+parser.add_option('-B', '--bloat', dest='show_bloat',
+ action='store_true', default=False,
+ help='Show changes in function code size for each board')
+parser.add_option('-c', '--count', dest='count', type='int',
+ default=-1, help='Run build on the top n commits')
+parser.add_option('-e', '--show_errors', action='store_true',
+ default=False, help='Show errors and warnings')
+parser.add_option('-f', '--force-build', dest='force_build',
+ action='store_true', default=False,
+ help='Force build of boards even if already built')
+parser.add_option('-d', '--detail', dest='show_detail',
+ action='store_true', default=False,
+ help='Show detailed information for each board in summary')
+parser.add_option('-g', '--git', type='string',
+ help='Git repo containing branch to build', default='.')
+parser.add_option('-H', '--full-help', action='store_true', dest='full_help',
+ default=False, help='Display the README file')
+parser.add_option('-j', '--jobs', dest='jobs', type='int',
+ default=None, help='Number of jobs to run at once (passed to make)')
+parser.add_option('-k', '--keep-outputs', action='store_true',
+ default=False, help='Keep all build output files (e.g. binaries)')
+parser.add_option('--list-tool-chains', action='store_true', default=False,
+ help='List available tool chains')
+parser.add_option('-n', '--dry-run', action='store_true', dest='dry_run',
+ default=False, help="Do a try run (describe actions, but no nothing)")
+parser.add_option('-Q', '--quick', action='store_true',
+ default=False, help='Do a rough build, with limited warning resolution')
+parser.add_option('-s', '--summary', action='store_true',
+ default=False, help='Show a build summary')
+parser.add_option('-S', '--show-sizes', action='store_true',
+ default=False, help='Show image size variation in summary')
+parser.add_option('--step', type='int',
+ default=1, help='Only build every n commits (0=just first and last)')
+parser.add_option('-t', '--test', action='store_true', dest='test',
+ default=False, help='run tests')
+parser.add_option('-T', '--threads', type='int',
+ default=None, help='Number of builder threads to use')
+parser.add_option('-u', '--show_unknown', action='store_true',
+ default=False, help='Show boards with unknown build result')
+
+parser.usage = """buildman -b <branch> [options]
+
+Build U-Boot for all commits in a branch. Use -n to do a dry run"""
+
+(options, args) = parser.parse_args()
+
+# Run our meagre tests
+if options.test:
+ RunTests()
+elif options.full_help:
+ pager = os.getenv('PAGER')
+ if not pager:
+ pager = 'more'
+ fname = os.path.join(os.path.dirname(sys.argv[0]), 'README')
+ command.Run(pager, fname)
+
+# Build selected commits for selected boards
+else:
+ control.DoBuildman(options, args)
--- /dev/null
+# Copyright (c) 2013 The Chromium OS Authors.
+#
+# See file CREDITS for list of people who contributed to this
+# project.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation; either version 2 of
+# the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+# MA 02111-1307 USA
+#
+
+import multiprocessing
+import os
+import sys
+
+import board
+import bsettings
+from builder import Builder
+import gitutil
+import patchstream
+import terminal
+import toolchain
+
+def GetPlural(count):
+ """Returns a plural 's' if count is not 1"""
+ return 's' if count != 1 else ''
+
+def GetActionSummary(is_summary, count, selected, options):
+ """Return a string summarising the intended action.
+
+ Returns:
+ Summary string.
+ """
+ count = (count + options.step - 1) / options.step
+ str = '%s %d commit%s for %d boards' % (
+ 'Summary of' if is_summary else 'Building', count, GetPlural(count),
+ len(selected))
+ str += ' (%d thread%s, %d job%s per thread)' % (options.threads,
+ GetPlural(options.threads), options.jobs, GetPlural(options.jobs))
+ return str
+
+def ShowActions(series, why_selected, boards_selected, builder, options):
+ """Display a list of actions that we would take, if not a dry run.
+
+ Args:
+ series: Series object
+ why_selected: Dictionary where each key is a buildman argument
+ provided by the user, and the value is the boards brought
+ in by that argument. For example, 'arm' might bring in
+ 400 boards, so in this case the key would be 'arm' and
+ the value would be a list of board names.
+ boards_selected: Dict of selected boards, key is target name,
+ value is Board object
+ builder: The builder that will be used to build the commits
+ options: Command line options object
+ """
+ col = terminal.Color()
+ print 'Dry run, so not doing much. But I would do this:'
+ print
+ print GetActionSummary(False, len(series.commits), boards_selected,
+ options)
+ print 'Build directory: %s' % builder.base_dir
+ for upto in range(0, len(series.commits), options.step):
+ commit = series.commits[upto]
+ print ' ', col.Color(col.YELLOW, commit.hash, bright=False),
+ print commit.subject
+ print
+ for arg in why_selected:
+ if arg != 'all':
+ print arg, ': %d boards' % why_selected[arg]
+ print ('Total boards to build for each commit: %d\n' %
+ why_selected['all'])
+
+def DoBuildman(options, args):
+ """The main control code for buildman
+
+ Args:
+ options: Command line options object
+ args: Command line arguments (list of strings)
+ """
+ gitutil.Setup()
+
+ bsettings.Setup()
+ options.git_dir = os.path.join(options.git, '.git')
+
+ toolchains = toolchain.Toolchains()
+ toolchains.Scan(options.list_tool_chains)
+ if options.list_tool_chains:
+ toolchains.List()
+ print
+ return
+
+ # Work out how many commits to build. We want to build everything on the
+ # branch. We also build the upstream commit as a control so we can see
+ # problems introduced by the first commit on the branch.
+ col = terminal.Color()
+ count = options.count
+ if count == -1:
+ if not options.branch:
+ str = 'Please use -b to specify a branch to build'
+ print col.Color(col.RED, str)
+ sys.exit(1)
+ count = gitutil.CountCommitsInBranch(options.git_dir, options.branch)
+ count += 1 # Build upstream commit also
+
+ if not count:
+ str = ("No commits found to process in branch '%s': "
+ "set branch's upstream or use -c flag" % options.branch)
+ print col.Color(col.RED, str)
+ sys.exit(1)
+
+ # Work out what subset of the boards we are building
+ boards = board.Boards()
+ boards.ReadBoards(os.path.join(options.git, 'boards.cfg'))
+ why_selected = boards.SelectBoards(args)
+ selected = boards.GetSelected()
+ if not len(selected):
+ print col.Color(col.RED, 'No matching boards found')
+ sys.exit(1)
+
+ # Read the metadata from the commits. First look at the upstream commit,
+ # then the ones in the branch. We would like to do something like
+ # upstream/master~..branch but that isn't possible if upstream/master is
+ # a merge commit (it will list all the commits that form part of the
+ # merge)
+ range_expr = gitutil.GetRangeInBranch(options.git_dir, options.branch)
+ upstream_commit = gitutil.GetUpstream(options.git_dir, options.branch)
+ series = patchstream.GetMetaDataForList(upstream_commit, options.git_dir,
+ 1)
+ series = patchstream.GetMetaDataForList(range_expr, options.git_dir, None,
+ series)
+
+ # By default we have one thread per CPU. But if there are not enough jobs
+ # we can have fewer threads and use a high '-j' value for make.
+ if not options.threads:
+ options.threads = min(multiprocessing.cpu_count(), len(selected))
+ if not options.jobs:
+ options.jobs = max(1, (multiprocessing.cpu_count() +
+ len(selected) - 1) / len(selected))
+
+ if not options.step:
+ options.step = len(series.commits) - 1
+
+ # Create a new builder with the selected options
+ output_dir = os.path.join('..', options.branch)
+ builder = Builder(toolchains, output_dir, options.git_dir,
+ options.threads, options.jobs, checkout=True,
+ show_unknown=options.show_unknown, step=options.step)
+ builder.force_config_on_failure = not options.quick
+
+ # For a dry run, just show our actions as a sanity check
+ if options.dry_run:
+ ShowActions(series, why_selected, selected, builder, options)
+ else:
+ builder.force_build = options.force_build
+
+ # Work out which boards to build
+ board_selected = boards.GetSelectedDict()
+
+ print GetActionSummary(options.summary, count, board_selected, options)
+
+ if options.summary:
+ # We can't show function sizes without board details at present
+ if options.show_bloat:
+ options.show_detail = True
+ builder.ShowSummary(series.commits, board_selected,
+ options.show_errors, options.show_sizes,
+ options.show_detail, options.show_bloat)
+ else:
+ builder.BuildBoards(series.commits, board_selected,
+ options.show_errors, options.keep_outputs)
--- /dev/null
+#
+# Copyright (c) 2012 The Chromium OS Authors.
+#
+# See file CREDITS for list of people who contributed to this
+# project.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation; either version 2 of
+# the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+# MA 02111-1307 USA
+#
+
+import os
+import shutil
+import sys
+import tempfile
+import time
+import unittest
+
+# Bring in the patman libraries
+our_path = os.path.dirname(os.path.realpath(__file__))
+sys.path.append(os.path.join(our_path, '../patman'))
+
+import board
+import bsettings
+import builder
+import control
+import command
+import commit
+import toolchain
+
+errors = [
+ '''main.c: In function 'main_loop':
+main.c:260:6: warning: unused variable 'joe' [-Wunused-variable]
+''',
+ '''main.c: In function 'main_loop':
+main.c:295:2: error: 'fred' undeclared (first use in this function)
+main.c:295:2: note: each undeclared identifier is reported only once for each function it appears in
+make[1]: *** [main.o] Error 1
+make: *** [common/libcommon.o] Error 2
+Make failed
+''',
+ '''main.c: In function 'main_loop':
+main.c:280:6: warning: unused variable 'mary' [-Wunused-variable]
+''',
+ '''powerpc-linux-ld: warning: dot moved backwards before `.bss'
+powerpc-linux-ld: warning: dot moved backwards before `.bss'
+powerpc-linux-ld: u-boot: section .text lma 0xfffc0000 overlaps previous sections
+powerpc-linux-ld: u-boot: section .rodata lma 0xfffef3ec overlaps previous sections
+powerpc-linux-ld: u-boot: section .reloc lma 0xffffa400 overlaps previous sections
+powerpc-linux-ld: u-boot: section .data lma 0xffffcd38 overlaps previous sections
+powerpc-linux-ld: u-boot: section .u_boot_cmd lma 0xffffeb40 overlaps previous sections
+powerpc-linux-ld: u-boot: section .bootpg lma 0xfffff198 overlaps previous sections
+'''
+]
+
+
+# hash, subject, return code, list of errors/warnings
+commits = [
+ ['1234', 'upstream/master, ok', 0, []],
+ ['5678', 'Second commit, a warning', 0, errors[0:1]],
+ ['9012', 'Third commit, error', 1, errors[0:2]],
+ ['3456', 'Fourth commit, warning', 0, [errors[0], errors[2]]],
+ ['7890', 'Fifth commit, link errors', 1, [errors[0], errors[3]]],
+ ['abcd', 'Sixth commit, fixes all errors', 0, []]
+]
+
+boards = [
+ ['board0', 'arm', 'armv7', 'ARM Board 1', 'Tester', '', ''],
+ ['board1', 'arm', 'armv7', 'ARM Board 2', 'Tester', '', ''],
+ ['board2', 'powerpc', 'powerpc', 'PowerPC board 1', 'Tester', '', ''],
+ ['board3', 'powerpc', 'mpc5xx', 'PowerPC board 2', 'Tester', '', ''],
+ ['board4', 'sandbox', 'sandbox', 'Sandbox board', 'Tester', '', '']
+]
+
+class Options:
+ """Class that holds build options"""
+ pass
+
+class TestBuild(unittest.TestCase):
+ """Test buildman
+
+ TODO: Write tests for the rest of the functionality
+ """
+ def setUp(self):
+ # Set up commits to build
+ self.commits = []
+ sequence = 0
+ for commit_info in commits:
+ comm = commit.Commit(commit_info[0])
+ comm.subject = commit_info[1]
+ comm.return_code = commit_info[2]
+ comm.error_list = commit_info[3]
+ comm.sequence = sequence
+ sequence += 1
+ self.commits.append(comm)
+
+ # Set up boards to build
+ self.boards = board.Boards()
+ for brd in boards:
+ self.boards.AddBoard(board.Board(*brd))
+ self.boards.SelectBoards([])
+
+ # Set up the toolchains
+ bsettings.Setup()
+ self.toolchains = toolchain.Toolchains()
+ self.toolchains.Add('arm-linux-gcc', test=False)
+ self.toolchains.Add('sparc-linux-gcc', test=False)
+ self.toolchains.Add('powerpc-linux-gcc', test=False)
+ self.toolchains.Add('gcc', test=False)
+
+ def Make(self, commit, brd, stage, *args, **kwargs):
+ result = command.CommandResult()
+ boardnum = int(brd.target[-1])
+ result.return_code = 0
+ result.stderr = ''
+ result.stdout = ('This is the test output for board %s, commit %s' %
+ (brd.target, commit.hash))
+ if boardnum >= 1 and boardnum >= commit.sequence:
+ result.return_code = commit.return_code
+ result.stderr = ''.join(commit.error_list)
+ if stage == 'build':
+ target_dir = None
+ for arg in args:
+ if arg.startswith('O='):
+ target_dir = arg[2:]
+
+ if not os.path.isdir(target_dir):
+ os.mkdir(target_dir)
+ #time.sleep(.2 + boardnum * .2)
+
+ result.combined = result.stdout + result.stderr
+ return result
+
+ def testBasic(self):
+ """Test basic builder operation"""
+ output_dir = tempfile.mkdtemp()
+ if not os.path.isdir(output_dir):
+ os.mkdir(output_dir)
+ build = builder.Builder(self.toolchains, output_dir, None, 1, 2,
+ checkout=False, show_unknown=False)
+ build.do_make = self.Make
+ board_selected = self.boards.GetSelectedDict()
+
+ #build.BuildCommits(self.commits, board_selected, False)
+ build.BuildBoards(self.commits, board_selected, False, False)
+ build.ShowSummary(self.commits, board_selected, True, False,
+ False, False)
+
+ def _testGit(self):
+ """Test basic builder operation by building a branch"""
+ base_dir = tempfile.mkdtemp()
+ if not os.path.isdir(base_dir):
+ os.mkdir(base_dir)
+ options = Options()
+ options.git = os.getcwd()
+ options.summary = False
+ options.jobs = None
+ options.dry_run = False
+ #options.git = os.path.join(base_dir, 'repo')
+ options.branch = 'test-buildman'
+ options.force_build = False
+ options.list_tool_chains = False
+ options.count = -1
+ options.git_dir = None
+ options.threads = None
+ options.show_unknown = False
+ options.quick = False
+ options.show_errors = False
+ options.keep_outputs = False
+ args = ['tegra20']
+ control.DoBuildman(options, args)
+
+if __name__ == "__main__":
+ unittest.main()
--- /dev/null
+# Copyright (c) 2012 The Chromium OS Authors.
+#
+# See file CREDITS for list of people who contributed to this
+# project.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation; either version 2 of
+# the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+# MA 02111-1307 USA
+#
+
+import glob
+import os
+
+import bsettings
+import command
+
+class Toolchain:
+ """A single toolchain
+
+ Public members:
+ gcc: Full path to C compiler
+ path: Directory path containing C compiler
+ cross: Cross compile string, e.g. 'arm-linux-'
+ arch: Architecture of toolchain as determined from the first
+ component of the filename. E.g. arm-linux-gcc becomes arm
+ """
+
+ def __init__(self, fname, test, verbose=False):
+ """Create a new toolchain object.
+
+ Args:
+ fname: Filename of the gcc component
+ test: True to run the toolchain to test it
+ """
+ self.gcc = fname
+ self.path = os.path.dirname(fname)
+ self.cross = os.path.basename(fname)[:-3]
+ pos = self.cross.find('-')
+ self.arch = self.cross[:pos] if pos != -1 else 'sandbox'
+
+ env = self.MakeEnvironment()
+
+ # As a basic sanity check, run the C compiler with --version
+ cmd = [fname, '--version']
+ if test:
+ result = command.RunPipe([cmd], capture=True, env=env)
+ self.ok = result.return_code == 0
+ if verbose:
+ print 'Tool chain test: ',
+ if self.ok:
+ print 'OK'
+ else:
+ print 'BAD'
+ print 'Command: ', cmd
+ print result.stdout
+ print result.stderr
+ else:
+ self.ok = True
+ self.priority = self.GetPriority(fname)
+
+ def GetPriority(self, fname):
+ """Return the priority of the toolchain.
+
+ Toolchains are ranked according to their suitability by their
+ filename prefix.
+
+ Args:
+ fname: Filename of toolchain
+ Returns:
+ Priority of toolchain, 0=highest, 20=lowest.
+ """
+ priority_list = ['-elf', '-unknown-linux-gnu', '-linux', '-elf',
+ '-none-linux-gnueabi', '-uclinux', '-none-eabi',
+ '-gentoo-linux-gnu', '-linux-gnueabi', '-le-linux', '-uclinux']
+ for prio in range(len(priority_list)):
+ if priority_list[prio] in fname:
+ return prio
+ return prio
+
+ def MakeEnvironment(self):
+ """Returns an environment for using the toolchain.
+
+ Thie takes the current environment, adds CROSS_COMPILE and
+ augments PATH so that the toolchain will operate correctly.
+ """
+ env = dict(os.environ)
+ env['CROSS_COMPILE'] = self.cross
+ env['PATH'] += (':' + self.path)
+ return env
+
+
+class Toolchains:
+ """Manage a list of toolchains for building U-Boot
+
+ We select one toolchain for each architecture type
+
+ Public members:
+ toolchains: Dict of Toolchain objects, keyed by architecture name
+ paths: List of paths to check for toolchains (may contain wildcards)
+ """
+
+ def __init__(self):
+ self.toolchains = {}
+ self.paths = []
+ for name, value in bsettings.GetItems('toolchain'):
+ if '*' in value:
+ self.paths += glob.glob(value)
+ else:
+ self.paths.append(value)
+
+
+ def Add(self, fname, test=True, verbose=False):
+ """Add a toolchain to our list
+
+ We select the given toolchain as our preferred one for its
+ architecture if it is a higher priority than the others.
+
+ Args:
+ fname: Filename of toolchain's gcc driver
+ test: True to run the toolchain to test it
+ """
+ toolchain = Toolchain(fname, test, verbose)
+ add_it = toolchain.ok
+ if toolchain.arch in self.toolchains:
+ add_it = (toolchain.priority <
+ self.toolchains[toolchain.arch].priority)
+ if add_it:
+ self.toolchains[toolchain.arch] = toolchain
+
+ def Scan(self, verbose):
+ """Scan for available toolchains and select the best for each arch.
+
+ We look for all the toolchains we can file, figure out the
+ architecture for each, and whether it works. Then we select the
+ highest priority toolchain for each arch.
+
+ Args:
+ verbose: True to print out progress information
+ """
+ if verbose: print 'Scanning for tool chains'
+ for path in self.paths:
+ if verbose: print " - scanning path '%s'" % path
+ for subdir in ['.', 'bin', 'usr/bin']:
+ dirname = os.path.join(path, subdir)
+ if verbose: print " - looking in '%s'" % dirname
+ for fname in glob.glob(dirname + '/*gcc'):
+ if verbose: print " - found '%s'" % fname
+ self.Add(fname, True, verbose)
+
+ def List(self):
+ """List out the selected toolchains for each architecture"""
+ print 'List of available toolchains (%d):' % len(self.toolchains)
+ if len(self.toolchains):
+ for key, value in sorted(self.toolchains.iteritems()):
+ print '%-10s: %s' % (key, value.gcc)
+ else:
+ print 'None'
+
+ def Select(self, arch):
+ """Returns the toolchain for a given architecture
+
+ Args:
+ args: Name of architecture (e.g. 'arm', 'ppc_8xx')
+
+ returns:
+ toolchain object, or None if none found
+ """
+ for name, value in bsettings.GetItems('toolchain-alias'):
+ if arch == name:
+ arch = value
+
+ if not arch in self.toolchains:
+ raise ValueError, ("No tool chain found for arch '%s'" % arch)
+ return self.toolchains[arch]
projects / u-boot.git / commitdiff
