Ownership of an unmaintained module with millions of weekly downloads passed on to a malicious actor (NPM event-stream)

Issue: https://github.com/dominictarr/event-stream/issues/116
Statement from the original main Dev: https://gist.github.com/dominictarr/9fd9c1024c94592bc7268d36b8d83b3a

The discussion also relates to the problem of maintainers of free software programs who are not paid for the maintenance, but I think that’s an entirely separate, distinct issue.

Interested in your thoughts here.

Edits:

More context and an interesting take: https://blog.tidelift.com/event-stream-100-million-downloads-unmaintained-hacked.-now-can-we-pay-the-maintainers

Good points: https://twitter.com/garybernhardt/status/1067112352296751104

Good factual recap and more info in the attack, by NPM: https://blog.npmjs.org/post/180565383195/details-about-the-event-stream-incident

1 Like

Issue: https://github.com/dominictarr/event-stream/issues/116
Statement from the original main Dev: statement on event-stream compromise · GitHub

The discussion also relates to the problem of maintainers of free software programs who are not paid for the maintenance, but I think that’s an entirely separate, distinct issue.

Hm. What’s the issue?

Sorry, I don’t feel like wading through a long thread in an issue
tracker (less so on github, even less so about NPM).

Interested in your thoughts here.

If my hunch is correct, I’d say this isn’t atypical of NPM. It
is less and less “free software” (the ideal) and more and more
“open source” (the business model). But I don’t know yet precisely
what I’m talking about (see above).

Cheers
– tomás

The issue (as summed up in the title) is that:

  • a malicious actor obtained ownership of a module distributed on a community repo
  • the module is quite popular (millions of installs per week)
  • the version on the repo included malware vs. the commit was apparently not appearing on the origin (here, GitHub).

This raises several questions:

  • how communities should govern the management of repos and packages/modules in their repos
    (e.g. Debian repos vs AUR vs NPM)

  • how maintainers should handle situations where they don’t want to continue maintaining something (in my opinion, transferring ownership to a random developer does not seem like a great idea for a module with millions of weekly installs. But it seems that’s what happened here.)

  • how developers and users of other packages should handle dependencies on other people’s packages

  • how do we ensure audits, monitoring, etc.

  • how do we mitigate and remediate situations like this when they happen.

What else? Thoughts?

The issue (as summed up in the title) is that:

  • a malicious actor obtained ownership of a module distributed on a community repo

Uh, oh. Well, this is a known anti-pattern.

  • the module is quite popular (millions of installs per week)

As npm goes… (and oh, it is probably worse, since a sizeable
part of those “millions” is… continuous integration systems,
which are ultimately deploying the thing semi-automatically).

  • the version on the repo included malware vs. the commit was apparently not appearing on the origin (here, GitHub).

Again, uh-oh.

This raises several questions:

  • how communities should govern the management of repos and packages/modules in their repos
    (e.g. Debian repos vs AUR vs NPM)

In the case of Debian I’m somewhat confident, because I trust its
culture. But it does pay a price for that – it’s perceived as
“slow”, and everyone and her dog is clamoring for setups like
Flatpak, where bleeding edge flows directly into end-user system.

Some software packages (esp. those which have received more power
than is healthy for them – think Web browsers) include an auto-update
system which bypasses the distribution’s.

Personally I sleep better by having a QA instance between the software
maker and myself (unless it is one of those rare packages I’m especially
interested in, and where I take the QA responsibility myself, but I
can afford that for a small handful of packages, at most).

  • how maintainers should handle situations where they don’t want to continue maintaining something (in my opinion, transferring ownership to a random developer does not seem like a great idea for a module with millions of weekly installs. But it seems that’s what happened here.)

There are several levels of maintainers: the software maker’s
maintainers, the distro package maintainers, etc.

Of course, we are seeing the same “rationalization pressures” so
dear to us in this modern consumer economy. And Free software (more
so when confused with Open Source) is “free”, meaning that you can
“start” your “up” without contributing – leading to an appalling
state of infrastructure.

Look up heartbleed and the OpenSSL libraries to see a sad story which
predates your npm story by some two years or so. It turned out that
the whole of OpenSSL, on which relied the Web’s security (banking and
all that) was kept running by (I believe) four persons, all of them
doing it in their free time.

The “solution” at that time was to create the Core Infrastructure
Initiative [1] where vendors could throw in some cash to keep
the lights on for important projects.

This is all well and good, but imagine diluting that over the (how
many? 450k?) packages in the npm repository. Hmmm.

My take on this? We are forgetting how to program and are compelled
to go shopping more and more often. But I’m an old curmudgeon myself.

  • how developers and users of other packages should handle dependencies on other people’s packages

Typically this is based on trust, and trust is based on
culture. A difficult thing in itself. I don’t thing there’s
a technical solution to that. It is an eminently social
problem.

  • how do we ensure audits, monitoring, etc.

How do we ensure that those profitting from free (software, data, etc.)
give back something? Take Amazon’s gift to the Wikimedia foundation.
They donated $1M, which sounds like a lot of money, until you learn
that Alexa wouldn’t have been the success it was without Wikipedia’s
huge and well-structured data basis.

We’re just seeing the same anti-pattern of strip-mining without regard
to externalities (environment, social repercussions, etc.) which we
know so well and love and which is the hallmark of extractive capitalism.
Surprised? I’m not.

  • how do we mitigate and remediate situations like this when they happen.

Perhaps by pursuing other ways of doing things. Cooperatives. Alternative
economy. Stay out of the way from Big Platforms (and their anti-patterns:
that’s why I’m so sceptical of this very Discourse thingy, btw.)

What else? Thoughts?

I was very excited by the last Bits und Bäume in Berlin: I think we
Free Software people should look around us for close relatives in
other fields.

Cheers

[1] https://www.coreinfrastructure.org/
– tomás

Thanks for bringing up this topic. It’s exciting (in positive and negative ways) but I don’t have any solution or insightful answer to your questions…

In general, I found this tweet by Gary Bernhardt the probably most fitting description of my thoughts:

There was an option 3: don’t decompose your application’s dependency graph into thousands of packages. People who argued that position were dismissed as (to paraphrase heavily) old and slow. That ship has sailed, and now we’re here.

Basically: NPM is screwed up, and there’s no simple solution.

I guess it’s a questions of thresholds the different repositories create. Debian has high standards, AUR lower ones but still some checks, and NPM seems to be open for almost any contributions. I feel that the messiness NPM bears proves that it’s undesirable to have such low contribution thresholds.

  • how maintainers should handle situations where they don’t want to continue maintaining something (in my opinion, transferring ownership to a random developer does not seem like a great idea for a module with millions of weekly installs. But it seems that’s what happened here.)

Well, these are individual decisions. Why do programmers even include dependencies on code authored by inactive maintainers? Well, because it’s simple, and perhaps such a dependency cannot even be detected on first sight.

Somehow this reminds me of Docker Hub and undermaintained packages.

No idea, really, especially with NPM. The simple answer for users would be “don’t use anything over NPM”. That’s how I deal with it since I do not feel competent enough to understand the dependency trees, other than those of typical AUR packages. But that might not be realistic for other users or organisations.

This story reminds me of this “essay” https://hackernoon.com/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5

It was published early 2018 and clearly foretold what happened here.