Node.js application dependency compilation rabbithole

predominant · June 14, 2018, 6:58am

I’m encountering a problem with an internal application. As such I won’t be able to post the package.json file that would be truly useful in debugging this. However, on a long dependency chain with npm, I’m seeing the following issues.

Its needing to compile some native dependencies by the looks.

First the plan.sh:

pkg_name=forest-portal
pkg_origin=rakops
pkg_version="0.1.0"
pkg_build_deps=(
  core/node
  core/git
  core/gcc
  core/glibc
  core/libpng
  core/make
  core/libtool
  core/autoconf
  core/automake
  core/nasm
)
pkg_deps=(
  core/coreutils
  core/nginx
)
pkg_svc_run="nginx -c ${pkg_svc_config_path}/nginx.conf"
pkg_svc_user="root"
pkg_exports=(
  [port]=port
)
pkg_exposes=(port)
pkg_binds_optional=(
  [api]="port"
)

do_build() {
  cp package*.json "${CACHE_PATH}/"
  pushd "${CACHE_PATH}" > /dev/null
  npm install \
    --unsafe-perm \
    --loglevel error \
    --fetch-retries 5
  fix_interpreter "node_modules/.bin/*" core/coreutils bin/env
  npm run-script build
  popd > /dev/null
}

do_install() {
  attach
  cp -r dist "${pkg_prefix}/"
}

And here are the build errors (from npm install):

> cwebp-bin@4.0.0 postinstall /hab/cache/src/forest-portal-0.1.0/node_modules/cwebp-bin
> node lib/install.js

  ⚠ spawn /hab/cache/src/forest-portal-0.1.0/node_modules/cwebp-bin/vendor/cwebp ENOENT
  ⚠ cwebp pre-build test failed
  ℹ compiling from source
  ✔ cwebp built successfully

> gifsicle@3.0.4 postinstall /hab/cache/src/forest-portal-0.1.0/node_modules/gifsicle
> node lib/install.js

  ⚠ spawn /hab/cache/src/forest-portal-0.1.0/node_modules/gifsicle/vendor/gifsicle ENOENT
  ⚠ gifsicle pre-build test failed
  ℹ compiling from source
  ✔ gifsicle built successfully

> mozjpeg@5.0.0 postinstall /hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg
> node lib/install.js

  ⚠ spawn /hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor/cjpeg ENOENT
  ⚠ mozjpeg pre-build test failed
  ℹ compiling from source
  ✖ Error: autoreconf -fiv && ./configure --disable-shared --disable-dependency-tracking --with-jpeg8  --prefix="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor" --bindir="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor" --libdir="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor" && make -j2 && make install -j2
Command failed: autoreconf -fiv
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force
autoreconf: configure.ac: tracing
autoreconf: configure.ac: not using Libtool
autoreconf: running: /hab/pkgs/core/autoconf/2.69/20170513214252/bin/autoconf --force
configure.ac:23: error: possibly undefined macro: AC_PROG_LIBTOOL
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
autoreconf: /hab/pkgs/core/autoconf/2.69/20170513214252/bin/autoconf failed with exit status: 1

    at ChildProcess.exithandler (child_process.js:275:12)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:925:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)

> optipng-bin@3.1.4 postinstall /hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin
> node lib/install.js

  ⚠ spawn /hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor/optipng ENOENT
  ⚠ optipng pre-build test failed
  ℹ compiling from source
  ✖ Error: ./configure --with-system-zlib --prefix="/hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor" --bindir="/hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor" && make install
Command failed: ./configure --with-system-zlib --prefix="/hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor" --bindir="/hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor"

    at ChildProcess.exithandler (child_process.js:275:12)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:925:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)

> pngquant-bin@4.0.0 postinstall /hab/cache/src/forest-portal-0.1.0/node_modules/pngquant-bin
> node lib/install.js

  ⚠ spawn /hab/cache/src/forest-portal-0.1.0/node_modules/pngquant-bin/vendor/pngquant ENOENT
  ⚠ pngquant pre-build test failed
  ℹ compiling from source
  ✔ pngquant pre-build test passed successfully
  ✖ Error: pngquant failed to build, make sure that libpng-dev is installed
    at Promise.all.then.arr (/hab/cache/src/forest-portal-0.1.0/node_modules/pngquant-bin/node_modules/bin-build/node_modules/execa/index.js:231:11)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! pngquant-bin@4.0.0 postinstall: `node lib/install.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the pngquant-bin@4.0.0 postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-06-14T06_43_24_580Z-debug.log

predominant · June 14, 2018, 7:17am

Further to this. Adding the following got me a little further:

do_prepare() {
  ACLOCAL_PATH="$(pkg_path_for core/libtool)/share/aclocal:${ACLOCAL_PATH}"
  export ACLOCAL_PATH
}

predominant · June 14, 2018, 7:18am

Now I am running into the following, and I don’t know how I can change the configure file to change that /usr/bin/file reference:

> mozjpeg@5.0.0 postinstall /hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg
> node lib/install.js

  ⚠ spawn /hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor/cjpeg ENOENT
  ⚠ mozjpeg pre-build test failed
  ℹ compiling from source
  ✖ Error: autoreconf -fiv && ./configure --disable-shared --disable-dependency-tracking --with-jpeg8  --prefix="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor" --bindir="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor" --libdir="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor" && make -j2 && make install -j2
Command failed: ./configure --disable-shared --disable-dependency-tracking --with-jpeg8  --prefix="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor" --bindir="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor" --libdir="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor"
./configure: line 8221: /usr/bin/file: No such file or directory
./configure: line 13605: PKG_PROG_PKG_CONFIG: command not found
./configure: line 13784: syntax error near unexpected token `libpng,'
./configure: line 13784: `PKG_CHECK_MODULES(libpng, libpng, HAVE_LIBPNG=1,'

    at ChildProcess.exithandler (child_process.js:275:12)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:925:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)

nellshamrell · June 14, 2018, 5:27pm

Thanks for reporting this! I’m taking a look now.

nellshamrell · June 14, 2018, 5:44pm

Are you using a template for your configure file? Or is it autogenerated? Static?

predominant · June 14, 2018, 10:24pm

The configure file comes from the mozjpeg dependency. That’s downloaded when npm install runs.

nellshamrell · June 14, 2018, 10:44pm

Gotcha, let me take another look.

smacfarlane · June 14, 2018, 10:57pm

One thing you can do when you encounter this scenario is create a symlink in the do_prepare callback and then remove it in the do_end callback.

You can see this pattern in action in the core/kmod package: https://github.com/habitat-sh/core-plans/blob/master/kmod/plan.sh#L18-L21

Another option is to use sed to replace the offending line with a full path to the habitat provided file call, as seen with the /bin/pwd call in this plan: https://github.com/habitat-sh/core-plans/blob/master/gnupg/plan.sh#L21-L22

predominant · June 15, 2018, 12:49am

That got me a step further @smacfarlane, thanks!

Here’s whats up next, syntax errors, which are likely I cannot solve.

  ⚠ spawn /hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor/cjpeg ENOENT
  ⚠ mozjpeg pre-build test failed
  ℹ compiling from source
  ✖ Error: autoreconf -fiv && ./configure --disable-shared --disable-dependency-tracking --with-jpeg8  --prefix="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor" --bindir="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor" --libdir="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor" && make -j2 && make install -j2
Command failed: ./configure --disable-shared --disable-dependency-tracking --with-jpeg8  --prefix="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor" --bindir="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor" --libdir="/hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor"
./configure: line 13605: PKG_PROG_PKG_CONFIG: command not found
./configure: line 13784: syntax error near unexpected token `libpng,'
./configure: line 13784: `PKG_CHECK_MODULES(libpng, libpng, HAVE_LIBPNG=1,'

    at ChildProcess.exithandler (child_process.js:275:12)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:925:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)

predominant · June 15, 2018, 12:50am

Its worth noting the plan changed a little to add dependencies on core/file and core/pkg-config

smacfarlane · June 15, 2018, 1:15am

I suspect you’ll need to modify ACLOCAL_PATH to include $(pkg_path_for core/pkg-config)/share/aclocal like you did for core/libtool

predominant · June 15, 2018, 2:45am

Absolutely right, thanks @smacfarlane

> mozjpeg@5.0.0 postinstall /hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg
> node lib/install.js

  ⚠ spawn /hab/cache/src/forest-portal-0.1.0/node_modules/mozjpeg/vendor/cjpeg ENOENT
  ⚠ mozjpeg pre-build test failed
  ℹ compiling from source
  ✔ mozjpeg built successfully

Next error that I see is from optipng

> optipng-bin@3.1.4 postinstall /hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin
> node lib/install.js

  ⚠ spawn /hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor/optipng ENOENT
  ⚠ optipng pre-build test failed
  ℹ compiling from source
  ✖ Error: ./configure --with-system-zlib --prefix="/hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor" --bindir="/hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor" && make install
Command failed: ./configure --with-system-zlib --prefix="/hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor" --bindir="/hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor"

    at ChildProcess.exithandler (child_process.js:275:12)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:925:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)

> pngquant-bin@4.0.0 postinstall /hab/cache/src/forest-portal-0.1.0/node_modules/pngquant-bin
> node lib/install.js

  ⚠ spawn /hab/cache/src/forest-portal-0.1.0/node_modules/pngquant-bin/vendor/pngquant ENOENT
  ⚠ pngquant pre-build test failed
  ℹ compiling from source
  ✔ pngquant pre-build test passed successfully
  ✖ Error: pngquant failed to build, make sure that libpng-dev is installed
    at Promise.all.then.arr (/hab/cache/src/forest-portal-0.1.0/node_modules/pngquant-bin/node_modules/bin-build/node_modules/execa/index.js:231:11)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

PS: Everyone who is helping, you folks are awesome

smacfarlane · June 15, 2018, 2:48pm

It looks like node is compiling some packages for you because it couldn’t find local dependencies. Either node, or the config/make of the library being built, is more than likely manipulating one of the ENV vars that lets your app find the necessary libraries and headers during the build process.

I would see if you can generate .harts for optipng and pngquant, and add those to your pkg_deps. That’s probably what you want, rather than having your package build them through npm every time you rebuild.

predominant · June 21, 2018, 4:21am

Not sure if there is a better way. I’ve tried with gifsicle, a dependency I need, and would love for it to be available without compiling from source.

pkg_deps=(core/gifsicle)
do_prepare() {
  hab pkg binlink core/gifsicle gifsicle -d "${CACHE_PATH}/node_modules/gifsicle/vendor"
}

I’ve tried with and without the do_prepare part.

The install seems to check the ability to run a command with --version options, but I can’t seem to coerce it to call the precompiled gifsicle:

github.com

imagemin/gifsicle-bin/blob/master/lib/install.js#L6


'use strict';
const binBuild = require('bin-build');
const log = require('logalot');
const bin = require('.');


bin.run(['--version'], err => {
	if (err) {
		log.warn(err.message);
		log.warn('gifsicle pre-build test failed');
		log.info('compiling from source');


		const cfg = [
			'./configure --disable-gifview --disable-gifdiff',
			`--prefix="${bin.dest()}" --bindir="${bin.dest()}"`
		].join(' ');

Latest sample output:

> gifsicle@3.0.4 postinstall /hab/cache/src/forest-portal-0.1.0/node_modules/gifsicle
> node lib/install.js

  ⚠ spawn /hab/cache/src/forest-portal-0.1.0/node_modules/gifsicle/vendor/gifsicle ENOENT
  ⚠ gifsicle pre-build test failed
  ℹ compiling from source
  ✔ gifsicle built successfully

smacfarlane · June 21, 2018, 2:36pm

This is an interesting case. It looks like the intent of the gifsicle module is to ensure that the binary is always present on the system, and does this by providing a pre-compiled version that is vendored, falling back to compilation if it can’t find an appropriate version.

This is problematic in the hab world, because the vendored gifsicle binary for linux-x86_64 has a package interpreter of /lib64/ld-linux-x86-64.so.2 which doesn’t exist in the studio.

If you fall back to letting it compile, I’m guessing you’ll need to add (at a minimum) core/autoconf, core/automake, and core/gcc to the pkg_build_deps, as well as core/glibc to the pkg_deps.

Habitat already gives you the “presence” guarantees of the gifsicle module though, so you could skip it as dependency and run gifsicle directly if you don’t need to support this app outside of a Habitat build.

predominant · June 22, 2018, 12:17am

gifsicle is the simple case. It would be nice if I could provide the binary instead of compiling, but the compilation works as is.

More complicated cases are the compilations that are continuing to fail:

> optipng-bin@3.1.4 postinstall /hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin
> node lib/install.js

  ⚠ spawn /hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor/optipng ENOENT
  ⚠ optipng pre-build test failed
  ℹ compiling from source
  ✖ Error: ./configure --with-system-zlib --prefix="/hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor" --bindir="/hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor" && make install
Command failed: ./configure --with-system-zlib --prefix="/hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor" --bindir="/hab/cache/src/forest-portal-0.1.0/node_modules/optipng-bin/vendor"

    at ChildProcess.exithandler (child_process.js:275:12)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:925:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)

> pngquant-bin@4.0.0 postinstall /hab/cache/src/forest-portal-0.1.0/node_modules/pngquant-bin
> node lib/install.js

  ⚠ spawn /hab/cache/src/forest-portal-0.1.0/node_modules/pngquant-bin/vendor/pngquant ENOENT
  ⚠ pngquant pre-build test failed
  ℹ compiling from source
  ✔ pngquant pre-build test passed successfully
  ✖ Error: pngquant failed to build, make sure that libpng-dev is installed
    at Promise.all.then.arr (/hab/cache/src/forest-portal-0.1.0/node_modules/pngquant-bin/node_modules/bin-build/node_modules/execa/index.js:231:11)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! pngquant-bin@4.0.0 postinstall: `node lib/install.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the pngquant-bin@4.0.0 postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-06-21T05_22_16_738Z-debug.log

predominant · June 25, 2018, 2:50am

Quick update. This is completely misleading. This message is output for any error during the compilation/make process.

Adding core/zlib to the build deps solved the optipng-bin installation.

Symlinking env into /usr/bin solved the pngquant-bin build:

do_before() {
  if [[ ! -f /usr/bin/file ]]; then
    hab pkg binlink core/file file -d /usr/bin
    hab pkg binlink core/busybox-static env -d /usr/bin
    _clean_file=true
  fi
}

Topic		Replies	Views
Tutorial: empty mytutorialapp-0.1.0.tar.gz file downloaded Habitat Help	0	872	July 28, 2016
Tutorial: empty mytutorialapp-0.1.0.tar.gz file downloaded Chef Habitat	0	639	July 28, 2016
Auto-Rebuild Changes Habitat Announcements	0	483	September 24, 2018
Can't execute binary within hab studio? Habitat Help	3	645	May 31, 2018
Post big rebuild build problems Habitat Help	9	506	June 19, 2018

Node.js application dependency compilation rabbithole

Related Topics