Emerge and Make the World!

Uncategorized Add comments

Emerge and Make the World! It sounds a bit like Genesis under duress! These are commands that marshal a heap of resources and generate a huge product. They’re not for the impatient.

A bit of a tangent, I had an order from Dell for 600 servers — it was a replication gig, so they all had to be the same. the first 350 or so were, but Dell changed suppliers for some of the components mid-stream, so we had 9.1GB drives mixed with 9.095GB drives from a different manufacturer, and a swapped out video driver. Where we needed “exact”, we got “same difference, right?”, and the replication had to be tailored for four platforms rather than one. Replication’s great for exactly identical supportable servers, but sometimes lacks flexibility.

How does that affect our Genesis? It means that the system is installed from pre-built, known-tested (binaries, testing the toolchain as well), pre-staged (faster download) content. It’s entirely repeatable, and easy to maintain and support if you’re willing to work on it longer than the 47 minutes it takes to replace the server (by LAN; faster using DVD).

Emerge is a tool used by Gentoo installers to interact with the Portage system to download, build, and install sourcecode for a new install. It downloads sourcecode, builds with whatever toolchain it has, and installs. It runs with elevated privilege, which is normal if we weren’t downloading foreign content. Installing 600 means 600 downloads of a few thousand software projects, but can adapt to mid-stream changes in HW. mid-stream changes in software versions means that you don’t get what the other guy got.

By contrast, RPM systems let you install the exact same binaries as the guy beside you, you can cache the pre-built binary payload, the dependency-table is known before the install begins, you can marshal requirements and make the change is a relatively short time.

It would be silly to install a second system of dependency-mapping that competes with the OS. Seriously, when you go to London, you drive on the left; when you’re on RedHat, track your dependencies in RPM — if your wheel is better, don’t reinvent, augment the one that’s there. Share your improvement. That’s the Stallman way, and the GPL, isn’t it?

“RPM” is like the prefabricated construction of the world.

“make World” was how X11 was built — Imake && make World. Imake was similar to qmake in that it needed the platform defined first in the tool before the portable Imakefiles could be converted to makefiles to run the build. Opensource turned to automake, autconf, libtool for unknown platforms for the chance to “figure out” the unknown platform before build starts, but the Embedded space and cross-compiling seems to favour Qt (qmake) for some embedded devices, BuildTool and OpenEmbedded for others.

“make World” took a long time.

SSO installs on system 5 rel 5 were like “custom” installs on system 3.2 rel 4.2 except that there was a huge design around concentric containers: Packages, Parcels, Containers, and Products. It added logical aggregates of packages as well as namespaces to the packaging world. It’s gone, we sued them in a situation similar to a GPL violation. Unfortunately, the technologies and innovation are lost, since we don’t read the court cases nor the ~23 years of development before that.

Ian Murdock gave his Linux variant a detailed specification for package quality. Debian Linux also gave us “apt”, which lets us one-command-install a server from binary packages, or get the dependencies of a source build to develop/extend a specific package. Reducing user-input reduces chances for human error.

Debian is like adding strict contracts for quality-appraisal to Prefabrication.

CPAN is a resource that can be harvested for yesterday’s latest perl: go to the ‘net, download the latest, build and install, running last night’s updated build scripts with permissions to trash a system or install malicious software. No problem, just update the source package tomorrow. No one will check… for datacenters, trusted systems, a known bill-of-materials (BOM) is needed before agreeing to do the work. I haven’t seen many datacenter change-requests go through with “I don’t know what it changes” and “I don’t know how much software it needs or how long it takes” as entries. CPAN is great for people who can rebuild a server if necessary.

CPAN is like when you need the latest tools, building methods, plumbers, framers, roofers, foundation work, landscapers, and you’ll just ship them raw materials and have it assembled onsite. It takes longer, and sometimes, conflicts break out at the soup-truck, but eventually, you’ll get the latest fusion of last week’s Tokyo and Paris fashion.

Dependencies change; installing 175 packages that change often gives a high probability of resulting in a new combination of all the versions of things. Tracking versions as part of dependencies is logistically easier with the Package/Component/Parcel system of SSOs and reduces the chance of collision (how many “httpd” are there? Cern::httpd and Apache::httpd differ). A BOM is something your system can check against currently-installed packages, and the user can also check. Binaries are already built, and others have used them before you get them, so there’s less risk of breakage if thousands have already installed it since it was released a week before.

Binaries built by others have less variance, therefore are not the most efficient for your precise hardware, risking 2-3% efficiency for a smaller set of combinations.

This is why I prefer RPM, CPM, SSO, DEB, even “custom” — a confirmable predictable BOM of pre-built components — over build-in-place systems. There a higher degree of deliberate behavior, a lower chance through an innocent patch-bump to break a critical system through indirectly escalating degree of change. Build-in-place takes a long time, and done twice will get different results, but usually builds a functional system and allows a basement hack to go forward if you have the time to wait for your environment to be built.

I’m sure you’ve heard these same arguments before.

2 Responses to “Emerge and Make the World!”

  1. Jeffrey Paul Says:

    You told me this years ago. I ignored you because build-in-place is awesome. Then one day I found myself responsible for 200 OS installs on the same architecture.

    Now I use RPM.


  2. allanc Says:

    Hey, thanks man. Sometimes, my gospel has merit, but even a stopped-clock shows the correct time twice daily…

Leave a Reply

WP Theme & Icons by N.Design Studio
Entries RSS Comments RSS Log in