Can't execute binary within hab studio?


#1

Hello,

I’m wanting to build Hashicorp Nomad into a Habitat plan, but I’m running into an issue when building my plan. For reference the plan can be found here.

Stepping into the plan, the unzipped file is the file that is expected, but ldd and bash report “file not found”:

[1][default:/src:0]# build 
   : Loading /src/plan.sh
   nomad: Plan loaded
   nomad: Validating plan metadata
   nomad: Using HAB_BIN=/hab/pkgs/core/hab/0.55.0/20180321220925/bin/hab for installs, signing, and hashing
   nomad: hab-plan-build setup
   nomad: Writing pre_build file
   nomad: Resolving build dependencies
» Installing core/unzip from channel 'stable'
→ Using core/unzip/6.0/20180310001951
★ Install of core/unzip/6.0/20180310001951 complete with 0 new packages installed.
   nomad: Resolved build dependency 'core/unzip' to /hab/pkgs/core/unzip/6.0/20180310001951
» Installing core/strace from channel 'stable'
→ Using core/strace/4.19/20171011085951
★ Install of core/strace/4.19/20171011085951 complete with 0 new packages installed.
   nomad: Resolved build dependency 'core/strace' to /hab/pkgs/core/strace/4.19/20171011085951
   nomad: Resolving run dependencies
» Installing core/linux from channel 'stable'
→ Using core/linux/4.11.1/20170807154945
★ Install of core/linux/4.11.1/20170807154945 complete with 0 new packages installed.
   nomad: Resolved dependency 'core/linux' to /hab/pkgs/core/linux/4.11.1/20170807154945
» Installing core/glibc from channel 'stable'
→ Using core/glibc/2.22/20170513201042
★ Install of core/glibc/2.22/20170513201042 complete with 0 new packages installed.
   nomad: Resolved dependency 'core/glibc' to /hab/pkgs/core/glibc/2.22/20170513201042
   nomad: Setting up environment
   nomad: Populating runtime environment from dependencies
   nomad: WARN No RUNTIME_ENVIRONMENT metadata file found for core/linux/4.11.1/20170807154945; falling back to PATH metadata
   nomad: WARN Final generated PATH for /hab/pkgs/core/linux/4.11.1/20170807154945: /hab/pkgs/core/glibc/2.22/20170513201042/bin
   nomad: WARN No RUNTIME_ENVIRONMENT metadata file found for core/glibc/2.22/20170513201042; falling back to PATH metadata
   nomad: WARN Final generated PATH for /hab/pkgs/core/glibc/2.22/20170513201042: /hab/pkgs/core/glibc/2.22/20170513201042/bin
   nomad: Populating buildtime environment from dependencies
   nomad: WARN No RUNTIME_ENVIRONMENT metadata file found for core/strace/4.19/20171011085951; falling back to PATH metadata
   nomad: WARN Final generated PATH for /hab/pkgs/core/strace/4.19/20171011085951: /hab/pkgs/core/strace/4.19/20171011085951/bin:/hab/pkgs/core/glibc/2.22/20170513201042/bin
   nomad: Layering runtime environment on top of system environment
   nomad: Value of PATH is /hab/pkgs/core/hab-plan-build/0.55.0/20180321222204/bin:/hab/pkgs/core/xz/5.2.2/20170513214327/bin:/hab/pkgs/core/glibc/2.22/20170513201042/bin:/hab/pkgs/core/wget/1.19.1/20171024102323/bin:/hab/pkgs/core/openssl/1.0.2l/20171014213633/bin:/hab/pkgs/core/libidn/1.32/20170513215043/bin:/hab/pkgs/core/unzip/6.0/20180301000848/bin:/hab/pkgs/core/bzip2/1.0.6/20170513212938/bin:/hab/pkgs/core/tar/1.29/20170513213607/bin:/hab/pkgs/core/attr/2.4.47/20170513213059/bin:/hab/pkgs/core/acl/2.2.52/20170513213108/bin:/hab/pkgs/core/sed/4.2.2/20170513213123/bin:/hab/pkgs/core/rq/0.9.2/20170612005822/bin:/hab/pkgs/core/hab/0.55.0/20180321220925/bin:/hab/pkgs/core/gzip/1.6/20170513214605/bin:/hab/pkgs/core/pcre/8.38/20170513213423/bin:/hab/pkgs/core/ncurses/6.0/20170513213009/bin:/hab/pkgs/core/less/481/20170513213936/bin:/hab/pkgs/core/grep/2.22/20170513213444/bin:/hab/pkgs/core/gawk/4.1.3/20170513213646/bin:/hab/pkgs/core/findutils/4.4.2/20170513214305/bin:/hab/pkgs/core/file/5.24/20170513201915/bin:/hab/pkgs/core/coreutils/8.25/20170513213226/bin:/hab/pkgs/core/libcap/2.24/20170513213120/bin:/hab/pkgs/core/binutils/2.25.1/20170513201927/bin:/hab/pkgs/core/bash/4.3.42/20170513213519/bin
   nomad: Layering buildtime environment on top of system environment
   nomad: Found previous file 'nomad-0.8.3_linux_amd64.zip', attempting to re-use
   nomad: Verifying nomad-0.8.3_linux_amd64.zip
   nomad: Checksum verified for nomad-0.8.3_linux_amd64.zip
   nomad: Using cached and verified 'nomad-0.8.3_linux_amd64.zip'
   nomad: Verifying nomad-0.8.3_linux_amd64.zip
   nomad: Checksum verified for nomad-0.8.3_linux_amd64.zip
   nomad: Clean the cache
Archive:  nomad-0.8.3_linux_amd64.zip
  inflating: nomad-0.8.3/nomad       

### Attaching to debugging session

From: /src/plan.sh @ line 38 :

    28: # pkg_exposes=(port ssl-port)
    29: 
    30: pkg_binds_optional=(
    31:   [consul]="host"
    32: )
    33: 
    34: # unzip our package into ${pkg_name}-${pkg_version} instead of just ${pkg_name}
    35: do_unpack() {
    36:   cd "${HAB_CACHE_SRC_PATH}" || exit
    37:   unzip ${pkg_filename} -d "${pkg_name}-${pkg_version}"
 => 38:   attach
    39: }
    40: 
    41: 
    42: # Since we have a precompiled binary, skip trying to compile it
    43: do_build() {
    44:   return 0
    45: }
    46: 
    47: do_install() {
    48:   attach

[1] nomad(do_unpack)> ls
nomad-0.8.3  nomad-0.8.3_linux_amd64.zip
[2] nomad(do_unpack)> ./nomad-0.8.3/nomad            
/hab/pkgs/core/hab-plan-build/0.55.0/20180321222204/bin/public.bash: line 240: ./nomad-0.8.3/nomad: No such file or directory
[3] nomad(do_unpack)> stat ./nomad-0.8.3/nomad 
  File: './nomad-0.8.3/nomad'
  Size: 44838240        Blocks: 87576      IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 1463464     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-05-27 00:14:42.700624230 +0000
Modify: 2018-04-27 17:55:55.000000000 +0000
Change: 2018-05-27 00:14:31.153624043 +0000
 Birth: -
[4] nomad(do_unpack)> file ./nomad-0.8.3/nomad 
./nomad-0.8.3/nomad: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=c55b7047bd313893aba2e9a880f1b3d8a89236a8, not stripped
[5] nomad(do_unpack)> ldd ./nomad-0.8.3/nomad 
/hab/pkgs/core/glibc/2.22/20170513201042/bin/ldd: line 117: ./nomad-0.8.3/nomad: No such file or directory
[6] nomad(do_unpack)> sha256sum nomad-0.8.3/nomad            
cab4e307bfd8d79091a503857ecde61ccc4ef0ecc54b88e8f9aa880a4f777720  nomad-0.8.3/nomad

Completing the build:

   nomad: hab-plan-build cleanup
   nomad: 
   nomad: Source Path: /hab/cache/src/nomad-0.8.3
   nomad: Installed Path: /hab/pkgs/qubitrenegade/nomad/0.8.3/20180527001426
   nomad: Artifact: /src/results/qubitrenegade-nomad-0.8.3-20180527001426-x86_64-linux.hart
   nomad: Build Report: /src/results/last_build.env
   nomad: SHA256 Checksum: ab84ffd66f889b524fbc9d62acce72d59ca6bece53528e4cfc1a1cd1e233d92c
   nomad: Blake2b Checksum: da393dfb0bc27128de40c12ecf82d3720d0654a1e209ec804080a6860b8cfa9a
   nomad: 
   nomad: I love it when a plan.sh comes together.
   nomad: 
   nomad: Build time: 3m43s

and installing the resulting package allows me to execute it “natively”:

$ cat results/last_build.env 
pkg_origin=qubitrenegade
pkg_name=nomad
pkg_version=0.8.3
pkg_release=20180527001426
pkg_ident=qubitrenegade/nomad/0.8.3/20180527001426
pkg_artifact=qubitrenegade-nomad-0.8.3-20180527001426-x86_64-linux.hart
pkg_sha256sum=ab84ffd66f889b524fbc9d62acce72d59ca6bece53528e4cfc1a1cd1e233d92c
pkg_blake2bsum=da393dfb0bc27128de40c12ecf82d3720d0654a1e209ec804080a6860b8cfa9a
$ source results/last_build.env
$ sudo hab pkg install results/$pkg_artifact
☛ Verifying qubitrenegade/nomad/0.8.3/20180527001426
→ Using core/glibc/2.22/20170513201042
→ Using core/linux-headers/4.3/20170513200956
→ Using core/linux/4.11.1/20170807154945
✓ Installed qubitrenegade/nomad/0.8.3/20180527001426
★ Install of qubitrenegade/nomad/0.8.3/20180527001426 complete with 1 new packages installed.
$ /hab/pkgs/$pkg_ident/bin/nomad -version
Nomad v0.8.3 (c85483da3471f4bd3a7c3de112e95f551071769f)
$ file /hab/pkgs/$pkg_ident/bin/nomad
/hab/pkgs/qubitrenegade/nomad/0.8.3/20180527001426/bin/nomad: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=c55b7047bd313893aba2e9a880f1b3d8a89236a8, stripped
$ stat /hab/pkgs/$pkg_ident/bin/nomad
  File: /hab/pkgs/qubitrenegade/nomad/0.8.3/20180527001426/bin/nomad
  Size: 27963864        Blocks: 54624      IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 1245079     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: unconfined_u:object_r:default_t:s0
Access: 2018-05-26 18:23:38.517632870 -0600
Modify: 2018-05-26 18:17:47.000000000 -0600
Change: 2018-05-26 18:23:23.381632626 -0600
 Birth: -
$ ldd /hab/pkgs/$pkg_ident/bin/nomad
        linux-vdso.so.1 (0x00007ffe9f9b8000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f285e044000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f285dc8e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f285e262000)
$ sha256sum /hab/pkgs/$pkg_ident/bin/nomad
474ee8364a368481297b628a7c6b480a7e22e380bf07697a24a55f0fe5b78beb  /hab/pkgs/qubitrenegade/nomad/0.8.3/20180527001426/bin/nomad

Interestingly the nomad binary seems to have a different sha256sum when in the hab studio?

This also extends to running in docker:

$ cat results/last_docker_export.env 
id=b925dab81524
name=qubitrenegade/nomad
tags=0.8.3-20180527001426,0.8.3,latest
name_tags=qubitrenegade/nomad:0.8.3-20180527001426,qubitrenegade/nomad:0.8.3,qubitrenegade/nomad:latest
$ source results/last_docker_export.env
$ docker run $name
hab-sup(MR): Supervisor Member-ID 07dd3a22398c416ba260dfd1cc7da89d
hab-sup(MR): Starting qubitrenegade/nomad
nomad.default(UCW): Watching user.toml
hab-sup(MR): Starting gossip-listener on 0.0.0.0:9638
hab-sup(MR): Starting http-gateway on 0.0.0.0:9631
nomad.default(HK): run, compiled to /hab/svc/nomad/hooks/run
nomad.default(HK): Hooks compiled
nomad.default(SR): Hooks recompiled
default(CF): Updated server.hcl 497f08225e5cd30e481b21ece767c4d09d0eaafc12640bcbe563a8d3a9e04cb6
nomad.default(SR): Configuration recompiled
nomad.default(SR): Initializing
nomad.default(SV): Starting service as user=root, group=root
nomad.default(O): + exec nomad agent -config-file=/hab/svc/nomad/config/server.hcl
nomad.default(O): /hab/svc/nomad/run: exec: line 1: nomad: not found
nomad.default(E): + exec
hab-launch(SV): Child for service 'nomad.default' with PID 153 exited with code exit code: 2
...
$ docker exec -it inspiring_aryabhata /bin/bash
/ # which nomad
/bin/nomad
 # stat /bin/nomad
  File: '/bin/nomad' -> '/hab/pkgs/qubitrenegade/nomad/0.8.3/20180527001426/bin/nomad'
  Size: 60              Blocks: 8          IO Block: 4096   symbolic link
Device: 9ch/156d        Inode: 2021518     Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-05-27 00:29:54.000000000
Modify: 2018-05-27 00:29:54.000000000
Change: 2018-05-27 00:29:58.000000000

/ # file /bin/nomad 
/bin/bash: file: not found
/ # ldd /bin/nomad
/bin/ldd: eval: line 1: /bin/nomad: not found

I’m kinda stumped as there’s nothing obvious to me that would indicate that the file doesn’t actually exist…

Could it be one of the shared libraries that’s missing? Most of the other Hashicorp products are static binaries, so I’m wondering if that has something to do with it.


#2

It looks like the ldd output is showing it linking to /lib64, which is incorrect. You might need to leverage patchelf to fix up the dynamic paths in this binary


#3

Ah ha! I knew I had read something about that bit I just couldn’t get there.

That’s exactly what I needed, thanks!


#4

In case you’re curious I think I actually have a branch out in core plans that adds Nomad that needed a little more testing before merge. It’s definitely out of date but there might be some prior art in there worth looking at: