Cannot re-enter studio on Ubuntu 18.04: dev/proc/etc filesystems not unmounting


#1

Hi friends, I’ve got a new dev workstation (physical machine, fresh Ubuntu 18.04 install) and am seeing some whacky problems that I don’t understand.

  1. I create a studio in /home/irving/src - this works fine, I see the typical and expected (?) filesystems get mounted
devpts on /hab/studios/home--irving--src/dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
proc on /hab/studios/home--irving--src/proc type proc (rw,relatime)
sysfs on /hab/studios/home--irving--src/sys type sysfs (rw,relatime)
tmpfs on /hab/studios/home--irving--src/run type tmpfs (rw,relatime)
tmpfs on /hab/studios/home--irving--src/var/run/docker.sock type tmpfs (rw,nosuid,noexec,relatime,size=3283068k,mode=755)
/dev/nvme0n1p3 on /hab/studios/home--irving--src/hab/cache/artifacts type xfs (rw,relatime,attr2,inode64,noquota)
/dev/nvme0n1p3 on /hab/studios/home--irving--src/src type xfs (rw,relatime,attr2,inode64,noquota)
  1. I exit the studio, all filesystems still stick around

  2. I re-enter the studio, and get an error very early:

$ RUST_LOG=debug hab studio enter
[2019-01-17T18:15:25Z DEBUG habitat_common::ui] UI { shell: Shell { input: InputStream { isatty: true }, out: OutputStream { coloring: Auto, isatty: true, is_colored(): true, supports_color(): true }, err: OutputStream { coloring: Auto, isatty: true, is_colored(): true, supports_color(): true } } }
[2019-01-17T18:15:25Z DEBUG hab::command::studio::enter] Setting HAB_CACHE_KEY_PATH=/home/irving/.hab/cache/keys
[2019-01-17T18:15:25Z DEBUG hab::command::studio::enter] Setting ARTIFACT_PATH=/home/irving/.hab/cache/artifacts
[2019-01-17T18:15:25Z DEBUG habitat_core::os::process::imp] Calling execvp(): ("/usr/bin/sudo") ["-p", "[sudo hab-studio] password for %u: ", "-E", "hab", "studio", "enter"]
[2019-01-17T18:15:25Z DEBUG habitat_common::ui] UI { shell: Shell { input: InputStream { isatty: true }, out: OutputStream { coloring: Auto, isatty: true, is_colored(): true, supports_color(): true }, err: OutputStream { coloring: Auto, isatty: true, is_colored(): true, supports_color(): true } } }
[2019-01-17T18:15:25Z DEBUG habitat_core::os::process::imp] Calling execvp(): ("/hab/pkgs/core/hab-studio/0.73.0/20190115010614/bin/hab-studio") ["enter"]
   hab-studio: Creating Studio at /hab/studios/home--irving--src (default)
mount: mounting proc on /hab/studios/home--irving--src/proc failed: Resource busy
  1. I attempt to clean up the mounted filesystems (seeing 15-16 of EACH one!), but dev refuses to unmount
$ grep hab /proc/mounts | awk '{print $2}' | sort | uniq | xargs -n1 sudo umount -f
umount: /hab/studios/home--irving--src/dev: target is busy.
  1. One time I did a hab studio rm and it ended up wiping out some/all of my /dev/ filesystem and temporarily hosing the box until I rebooted - this scared the crap out of me so I’m nervous about doing that again . :slight_smile:

Has anyone seen this before? what kinds of things can I do to troubleshoot this?


#2

Providing a tiny bit more information, I found the VERBOSE flag and it appears that cleanup_studio is silently failing:

1][default:/src:0]# exit
logout
+ cleanup_studio
+ kill_launcher
...
+ umount_fs -v /hab/studios/home--irving--src3/run
+ eval _mount_point=$2
+ _mount_point=/hab/studios/home--irving--src3/run
+ is_fs_mounted /hab/studios/home--irving--src3/run
+ _mount_point=/hab/studios/home--irving--src3/run
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox mount
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox grep -q on /hab/studios/home--irving--src3/run type
+ return 0
+ [ -z  ]
+ umount_fs -v /hab/studios/home--irving--src3/sys
+ eval _mount_point=$2
+ _mount_point=/hab/studios/home--irving--src3/sys
+ is_fs_mounted /hab/studios/home--irving--src3/sys
+ _mount_point=/hab/studios/home--irving--src3/sys
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox mount
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox grep -q on /hab/studios/home--irving--src3/sys type
+ return 0
+ umount_fs -v /hab/studios/home--irving--src3/proc
+ eval _mount_point=$2
+ _mount_point=/hab/studios/home--irving--src3/proc
+ is_fs_mounted /hab/studios/home--irving--src3/proc
+ _mount_point=/hab/studios/home--irving--src3/proc
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox mount
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox grep -q on /hab/studios/home--irving--src3/proc type
+ return 0
+ umount_fs -v /hab/studios/home--irving--src3/dev/pts
+ eval _mount_point=$2
+ _mount_point=/hab/studios/home--irving--src3/dev/pts
+ is_fs_mounted /hab/studios/home--irving--src3/dev/pts
+ _mount_point=/hab/studios/home--irving--src3/dev/pts
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox mount
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox grep -q on /hab/studios/home--irving--src3/dev/pts type
+ return 0
+ umount_fs -v -l /hab/studios/home--irving--src3/dev
+ eval _mount_point=$3
+ _mount_point=/hab/studios/home--irving--src3/dev
+ is_fs_mounted /hab/studios/home--irving--src3/dev
+ _mount_point=/hab/studios/home--irving--src3/dev
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox mount
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox grep -q on /hab/studios/home--irving--src3/dev type
+ return 0
+ umount_fs -v -l /hab/studios/home--irving--src3/var/run/docker.sock
+ eval _mount_point=$3
+ _mount_point=/hab/studios/home--irving--src3/var/run/docker.sock
+ is_fs_mounted /hab/studios/home--irving--src3/var/run/docker.sock
+ _mount_point=/hab/studios/home--irving--src3/var/run/docker.sock
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox mount
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox grep -q on /hab/studios/home--irving--src3/var/run/docker.sock type
+ return 0

but they are clearly still mounted.