Foreman » Installer

You're right we don't deal with this well. Technically we've never supported going from Foreman to Katello. It's been on the wish list for a long time, but it's better to start fresh.

I believe the specific error you're seeing is #25657 and I tried to fix it once, but there were some challenges and I never got back to it.

Ewoud Kohl van Wijngaarden wrote:

You're right we don't deal with this well. Technically we've never supported going from Foreman to Katello. It's been on the wish list for a long time, but it's better to start fresh.

I believe the specific error you're seeing is #25657 and I tried to fix it once, but there were some challenges and I never got back to it.

I've gone through a couple of iterations of trying "to start fresh", but apparently I haven't unwound things far enough. The installer lays down LOTS of packages and files, and I don't know to what extent those need to be removed to avoid the problem. Does foreman-install not have an "uninstall" option? Bug #25657 talks about the need to "clear the global registry"; how can this be done? Uninstalling all packages with "foreman" in their name does not remove the problem.

No, in the age of virtual machines there hasn't really been a need to remove it: installing a fresh machine is often easier. There is `--reset-data` as an option to wipe the data but I'd really start with a fresh machine. What you're trying to do (upgrade from non-Katello to Katello) is simply not supported. Advanced users have been known to make it work, but they knew exactly what they were doing.

Ewoud Kohl van Wijngaarden wrote:

No, in the age of virtual machines there hasn't really been a need to remove it: installing a fresh machine is often easier. There is `--reset-data` as an option to wipe the data but I'd really start with a fresh machine. What you're trying to do (upgrade from non-Katello to Katello) is simply not supported. Advanced users have been known to make it work, but they knew exactly what they were doing.

I have to say I'm disappointed in that answer. We're not using a VM, and the machine is running lots of other services and test scenarios so re-installing the entire machine simply isn't an option. All modern package management facilities offer full support for cleanly uninstalling what has been installed, so implementing an install procedure that does not utilize this capability in favor of simply reinstalling the entire machine reminds me of the bad old days when we were told that "reinstalling Windows" was the only option...

I'm also surprised that you're calling this an "upgrade from non-Katello to Katello". My reading of the Foreman documentation describes Katello as a standard Foreman plugin, so I'd expect it to be something that could be installed and uninstalled as required. I don't mean to be disrespectful, but I guess my expectations are higher than what's reasonable for this product.

I may not be an "advanced user" (yet, though advanced in my years of experience I can certainly claim), but I will continue down my path of trying to reset the machine back to a point where a full install of Foreman with Katello can be achieved. If I succeed I will report back here with my findings. Thanks.

Steve Turner wrote:

Ewoud Kohl van Wijngaarden wrote:

No, in the age of virtual machines there hasn't really been a need to remove it: installing a fresh machine is often easier. There is `--reset-data` as an option to wipe the data but I'd really start with a fresh machine. What you're trying to do (upgrade from non-Katello to Katello) is simply not supported. Advanced users have been known to make it work, but they knew exactly what they were doing.

I have to say I'm disappointed in that answer. We're not using a VM, and the machine is running lots of other services and test scenarios so re-installing the entire machine simply isn't an option. All modern package management facilities offer full support for cleanly uninstalling what has been installed, so implementing an install procedure that does not utilize this capability in favor of simply reinstalling the entire machine reminds me of the bad old days when we were told that "reinstalling Windows" was the only option...

Our installer takes control over many aspects of the system. It is much more than "package management". For example, it reconfigures Apache. How do you imagine an "uninstall" works? Simply purge Apache? Reset it to a default state? Leave everything in place?

The documentation literally states that running it on an existing system is not advisable. The last sentence of the first paragraph on https://theforeman.org/manuals/3.4/index.html#2.1Installation is clear about that:

It’s not advisable to follow the steps below on an existing system, since the installer will affect the configuration of several components.

Steve Turner wrote:

I'm also surprised that you're calling this an "upgrade from non-Katello to Katello". My reading of the Foreman documentation describes Katello as a standard Foreman plugin, so I'd expect it to be something that could be installed and uninstalled as required. I don't mean to be disrespectful, but I guess my expectations are higher than what's reasonable for this product.

We call this out at the start of the installation chapter: https://theforeman.org/manuals/3.4/index.html#2.1Installation has a blue block warning about this:

If you want to manage content (for example, RPMs, Kickstart trees, ISO and KVM images, OSTree content, and more) with Foreman please follow the Katello installation instructions. If you are a new user, ensure that you familiarize yourself with Katello and decide whether you want to use its features because installing Katello on top of an existing Foreman is unsupported. Note that you cannot install Katello on Debian systems.

Perhaps you can give a suggestion on how this could be made clearer.

Ewoud Kohl van Wijngaarden wrote:

Our installer takes control over many aspects of the system. It is much more than "package management". For example, it reconfigures Apache. How do you imagine an "uninstall" works? Simply purge Apache? Reset it to a default state? Leave everything in place?

Of course, I don't know how deep your requirements are for configuring Apache, but I've done a significant amount of that myself and I've discovered that in every case where I had modified the Apache configuration directly, I didn't need to do so, instead taking advantage of the "drop-in" directories (e.g., /etc/httpd/conf.d) where configuration changes are simply added or removed as needed.

The documentation literally states that running it on an existing system is not advisable. The last sentence of the first paragraph on https://theforeman.org/manuals/3.4/index.html#2.1Installation is clear about that:

It’s not advisable to follow the steps below on an existing system, since the installer will affect the configuration of several components.

I confess to not being too concerned about that since this particular system is a test environment for incrementally learning about the operation of Foreman, Katello, Pulp, and various other components that lead up to what eventually comprises a full Red Hat Satellite environment. We are a large RHS shop, and keeping these systems healthy is an ongoing challenge (even with the support of Red Hat), so we feel it's to our advantage to learn as much about the underlying subsystems as possible.

Steve Turner wrote:

I'm also surprised that you're calling this an "upgrade from non-Katello to Katello". My reading of the Foreman documentation describes Katello as a standard Foreman plugin, so I'd expect it to be something that could be installed and uninstalled as required. I don't mean to be disrespectful, but I guess my expectations are higher than what's reasonable for this product.

We call this out at the start of the installation chapter: https://theforeman.org/manuals/3.4/index.html#2.1Installation has a blue block warning about this:

If you want to manage content (for example, RPMs, Kickstart trees, ISO and KVM images, OSTree content, and more) with Foreman please follow the Katello installation instructions. If you are a new user, ensure that you familiarize yourself with Katello and decide whether you want to use its features because installing Katello on top of an existing Foreman is unsupported. Note that you cannot install Katello on Debian systems.

Perhaps you can give a suggestion on how this could be made clearer.

Hah, well as I'm sure you're well aware, inexperienced users can "outsmart" even the clearest instructions and documentation. :-) I literally did not see the words "because installing Katello on top of an existing Foreman is unsupported", or if I did I didn't know what it meant because I didn't understand what each of those components do. It was only after I brought up the initial Foreman user interface and it did not include the facilities for managing those features that I realized I needed Katello.

As far as suggestions for how that could be made clearer, I'm not sure what to say. For me, to "ensure that you familiarize yourself with Katello and decide whether you want to use its features" would mean to run the system both with it and without it to see what it brings to the table. This is why I wanted to install the system in an incremental fashion in the first place, to understand it from the ground up. RHS is a highly complex system, and I need all the help I can get understanding what all the parts and pieces do.

Thanks for the help. I may be learning the hard way, but I AM learning. :-)

Steve Turner wrote:

Ewoud Kohl van Wijngaarden wrote:

Our installer takes control over many aspects of the system. It is much more than "package management". For example, it reconfigures Apache. How do you imagine an "uninstall" works? Simply purge Apache? Reset it to a default state? Leave everything in place?

Of course, I don't know how deep your requirements are for configuring Apache, but I've done a significant amount of that myself and I've discovered that in every case where I had modified the Apache configuration directly, I didn't need to do so, instead taking advantage of the "drop-in" directories (e.g., /etc/httpd/conf.d) where configuration changes are simply added or removed as needed.

By default we purge all unrelated configs and 100% manage it. This allows us to ensure it's compatible but also optimized. For example, we use the event mpm but that's incompatible with mod_php. Using the event mpm also allows us to enable HTTP/2 (Foreman 3.5 feature). That's just one example.

These are all trade offs, but we rarely get the request about an uninstall feature. I don't recall discussing it more than a handful of times in the decade I've been involved in the project.

If you want to force a clean slate, what probably works is:

rm /etc/foreman-installer/scenarios.d/last_scenario.yaml
dnf install foreman-installer-katello
foreman-installer --scenario katello --reset-data

Ewoud Kohl van Wijngaarden wrote:

These are all trade offs, but we rarely get the request about an uninstall feature. I don't recall discussing it more than a handful of times in the decade I've been involved in the project.

Fair enough; thank you for listening to me rant about it. :-)

If you want to force a clean slate, what probably works is:
[...]

Unfortunately, I already manually cleaned the system to the point where the re-install was able to succeed before I had a chance to try the --reset-data option. I'm curious if it would have helped with my situation, but at this point I'm not willing to trash the progress I've made to find out.

I mentioned that I would report back here with the steps I took to get things cleaned up, and I want to do that because there's nothing more frustrating than to find that one discussion where somebody was having the very same rare problem as you, but there is never any follow-up to let you know how they fixed it!

I started off by viewing the output of "dnf history" to see what packages it installed (which unfortunately does not show you all the dependencies that get pulled in), and in my first attempts I simply removed every package containing the word "foreman". That didn't work, so I repeated that and also removed any "puppet" packages:

dnf erase $(rpm -qa | grep foreman | sed -e "s/-[0-9].*//")
dnf erase puppet7-release puppet-agent

That still didn't work, so I removed everything else I could see that got installed after the time when I started the initial install (except Apache; I didn't suspect that to be a factor in the error):

dnf erase $(rpm -qa | grep ruby | sed -e "s/-[0-9].*//")
dnf erase redis
dnf erase postgresql-server
dnf erase postgresql

I also removed all the directories that the uninstall step for the packages didn't remove:

rm -rf /etc/foreman
rm -rf /etc/foreman-installer/
rm -rf /etc/foreman-proxy/
rm -rf /etc/puppetlabs/
rm -rf /var/lib/pgsql/

I suspect it was probably the removal of all the ruby content that fixed the problem, but I also removed all traces of postgresql in case that's where the "global registry" was implemented.

Hopefully this will be of help to that one misguided person who comes along three years from now with the same plight.

As for the original error that prompted me to open this issue: I'll let you decide how to proceed from here, if at all. Thanks.