The RubyGems “security incident”
Ruby Central posted an extremely concerning “Incident Response Timeline” today, in which they make a number of exaggerated or purely misleading claims. Here’s my effort to set the record straight.
First, and most importantly: I was a primary operator of RubyGems.org, securely and successfully, for over ten years. Ruby Central does not accuse me of any harms or damages in their post, in fact stating “we have no evidence to indicate that any RubyGems.org data was copied or retained by unauthorized parties, including Mr. Arko.”
The actions I took during a time of great confusion and uncertainty (created by Ruby Central!) were careful, specific, and aimed to defend both Ruby Central the organization and RubyGems.org the service from potential threats.
The majority of the team, including developers in the middle of paid full-time work for Ruby Central, had just had all of their permissions on GitHub revoked. And then restored six days later. And then revoked again the next day. Even after the second mass-deletion of team permissions, Marty Haught sent an email to the team within minutes, at 12:47pm PDT, saying he was (direct quote) “terribly sorry” and “I messed up”. Update: Added email timestamp.
The erratic and contradictory communication supplied by Marty Haught, and the complete silence from Shan and the board, made it impossible to tell exactly who had been authorized to take what actions. As this situation occurred, I was the primary on-call. My contractual, paid responsibility to Ruby Central was to defend the RubyGems.org service against potential threats.
Marty’s final email clearly stated “I’ll follow up more on this and engage with the governance rfc in good faith.”. Just a few minutes after that email, at 1:01pm PDT, Marty also posted a public GitHub comment, where he agreed to participate in the proposed governance process and stated “I’m committed to find the right governance model that works for us all. More to come.” Update: screenshot of comment removed and replaced with link, since the comment appears to still be visible (at least to logged out users) on GitHub.
Given Marty’s claims, the sudden permission deletions made no sense. Worried about the possibility of hacked accounts or some sort of social engineering, I took action as the primary on-call engineer to lock down the AWS account and prevent any actions by possible attackers. I did not change the email addresses on any accounts, leaving them all owned by a team-shared email at rubycentral.org, to ensure the organization retained overall control of the accounts, even if individuals were somehow taking unauthorized actions.
Within a couple of days, Ruby Central made an (unsigned) public statement, and various board members agreed to talk directly to maintainers. At that point, I realized that what I thought might have been a malicious takeover was both legitimate and deliberate, and Marty would never “fix the permissions structure”, or “follow up more” as he said.
Once I understood the situation, I backed off to let Ruby Central take care of their “security audit”. I left all accounts in a state where they could recover access. I did not alter, or try to alter, anything in the Ruby Central systems or GitHub repository after that. I was confident, at the time, that Ruby Central’s security experts would quickly remove all outside access.
... continue reading