Cleaned up LDFLAGS settings

Fixed issue with "--with-zlib" wiping out LDFLAGS