flawed study

Story: Flaws found in BSD, Linux software updatersTotal Replies: 8
Author Content
hughesjr

Jul 15, 2008
4:05 AM EDT
I would like to point out that the linked study is not accurate, at least for CentOS. Being listed as a "Public Mirror" for CentOS does NOT mean that users are directed to that mirror for yum updates.

CentOS has a script that we use to validate "Public Mirrors" before we add them to the CentOS "Yum Mirrorlists". This validation process downloads the metadata XML file and compares it against the Master CentOS Server, if the files don't match (usually because a mirror is waiting for an update and has not synced yet, but could also happen if fake metadata is generated) then that mirror is removed from the "Yum Mirrorlists" and users do not get updates from there. It will stay listed as a "Public Mirror" for up to 24 hours if outdated, but will be removed from the "Yum Mirrorlists" in 2-3 hours.

This study did not mention (and probably did not know) that this check is even in the CentOS update process.

I don't know if any other distros use something similar or not, but at least for CentOS they got a major part of it wrong. Other distros with similar policies should also speak out.

I have a detailed write up (for those interested) here:

http://www.hughesjr.com/content/view/22/2/
Sander_Marechal

Jul 15, 2008
4:49 AM EDT
I think their writeup about debian seems to be flawed as well. Debian signs all it's packages and repository indexes. If you would download an altered package from a malicious mirror, apt would start yelling and throwing kitchen utensils at you. Either because the signature does not match the content or because the signature is unknown.

From what I gather at a quick read, the authors of the study merely managed to get their server listed as a mirror. They did not investigate what would happen if they started to offer altered packages.
krisum

Jul 15, 2008
11:34 AM EDT
Sander,
Quoting: Either because the signature does not match the content or because the signature is unknown.
The main attack they are talking of is providing valid and properly signed but older versions of the packages (e.g. older debian's openssl) making apt believe that the older package is the latest one and then attacking the known vulnerability in that package. Unfortunately since most users will only rely on automatic updates without being conscious of the security bulletins, the only hurdle to such an attack is mirror checking as hughesjr points out for CentOS. I have no idea if Debian has anything similar though I always use [url=http://ftp.{country-code}.debian.org]http://ftp.{country-code}.debian.org[/url] kind of mirrors for precisely these kinds of issues.

Hughesjr, I have a question from the writeup on your site. In mirror control attack section you say:

Quoting: Even IF they did that, they do not have packages signed by a centos.org key. Because of our repomd.xml file checking, the likelyhood of this attack (as with the first one) is almost 0.
Why do they not have packages signed by a centos.org key? The old (and vulnerable) centos.org packages are public so they do have those older versions of packages that are properly signed. So I think that your mirror checking system is also vulnerable (due to precisely the reason you give in the article viz. the fake mirror can provide correct repomd data to your mirror checking scripts while providing older data to other systems). It is somewhat more work but does not negate the basic form of attack and the primary reason is that CentOS's mirror checking is, I believe, for testing outdated mirrors not for checking malicious mirrors.

I think that to really defeat this there are two possible approaches: 1) To use metadata signing (debian already does that) with expiration info in metadata as the authors say. This seems a little tricky though, since there is no fixed expiration for packages. Some packages may not get updated for years, while others may need frequent updates. I may be missing something here. 2) The simpler approach could be to always force clients to fetch repomd data (or Packages.bz2 in case of Debian) from known good mirrors only, while the packages can be fetched from any mirror.
jdixon

Jul 15, 2008
11:53 AM EDT
Well, I had to go refresh my memory about slacktool. It looks like it was released and last updated in 2001. :(

Needless to say, it's not part of the official Slackware release. Now, if they had listed slackpkg I'd be more concerned. They would even have a valid point with slapt-get/gslapt, since so many people use them. But slacktool???
hughesjr

Jul 15, 2008
3:15 PM EDT
krisum - IF they have old metadata (or modified metatdata) they are not serving updates to centos machines via yum for more than 2-3 hours ... even if they are using some type of DNS poisioning or man-in-the-middle attack (since our testing machines would get the bad repomd.xml file, we would flag it).

I don't think delaying updates for 2-3 hours will have any impact UNLESS they are able to put an unsigned file on your machine, which they are not able to do. That is what my comment is talking about.

Take a look at this link from your machine:

http://mirrorlist.centos.org/?arch=i386&release=5&repo=os

That list is 10 machines (yum picks a random one) ... the list is regenerated every 2-3 hours ... only 10/200 servers are listed ... mirrors with metadata that does not match the master server are not listed ... replace repo=os with repo=updates ... they are a separate list. With only 10 mirrors (randomly selected) getting on that list, even if they were able to give us good metadata and give bad metadata to someone else, they have a 1/20 chance of being on the list ... then a 1/10 chance after that of being selected.

There is no way (using the default config) that even a good mirror is going to be consistently on those lists, certainly not a mirror with bad metadata.

So, IF someone could provide YOU with old metadata (but not the centos test servers or they would be removed) ... IF they remained the only mirror you got updates from (highly unlikely unless you picked only one mirror) ... then they could prevent you from getting security updates.
hughesjr

Jul 15, 2008
3:24 PM EDT
OH ... and I am all for metadata signing ... but yum does not currently do that
krisum

Jul 16, 2008
12:46 AM EDT
hughesjr, At least in Fedora I had noticed that the list was generated depending on the geographical location of the client machine and remained more or less constant even after refreshes for quite a long time -- I may be wrong, though. Assuming that the list is refreshed with random mirrors every 2-3hrs, the *default* configuration in CentOS/Fedora will mostly not be vulnerable to this kind of attack. For that matter the default configuration in Debian/Ubuntu/... is also not vulnerable to this kind of attacks since they use the standard repo locations by default. This attack comes into play when a user chooses a particular mirror (e.g. because it looks like the fastest to him/her) believing it to be a good one. In Fedora too I used to chose a particular mirror because the mirror list generated from my location had mostly slow mirrors or sometimes unreachable from my machine causing yum to get stuck. To avoid the mentioned attacks for these situations I still think that a good solution will be have package managers get repo data only from known good mirrors (which is normally not configurable) and use mirrors for the signed packages (which is configurable in yum configuration).
hughesjr

Jul 16, 2008
4:58 AM EDT
krisum,

Our mirrorlist is also geoip based ... so for some places, close mirrors will appear consistently on the list, though, if they have bad metadata they will not be on the list at all.

But because of our "close servers" implementation, we will have more than 10 show up everywhere. So for some locations, a particular mirror might be in the list more often than in other locations (the US will have a different mirrorlist all the time, for example, while some other locations might get one or two on the list all the time).

Don't get me wrong, I am all for signed metadata. We test the repomd.xml files directly because we can't sign the metadata.

WRT only one mirror, if the user picks 2 or 3 instead of 1 and then uses the random (default) method for yum to pick mirrors, then they will still not need to worry about delayed updates. the only way the delayed updates is really a problem is if someone picks only one mirror. I would never do or recommend that.

The study also DOES have some valid points for getting package managers better, it just insinuates that there is a MAJOR PROBLEM now, which is an exaggeration IMHO.

krisum

Jul 16, 2008
10:02 AM EDT
hughesjr,

Quoting: The study also DOES have some valid points for getting package managers better, it just insinuates that there is a MAJOR PROBLEM now, which is an exaggeration IMHO.
I too think that it is a big exaggeration. Normal desktop users do not explicitly choose mirrors and just use whatever got set during installation. Those who do explicitly choose mirrors would be expected to verify at the very least that the chosen mirror gets updated frequently.

Quoting: Don't get me wrong, I am all for signed metadata. We test the repomd.xml files directly because we can't sign the metadata.
As I have mentioned rather than signing the metadata (or in addition to), I would favour separation of mirrors for repomd data and those for packages (the latter will be normally be a superset of former). Users then only get to configure where they get the (signed) packages from, and the package manager contacts only the fixed set of standard mirrors for repomd data which is thus known to be good as well as updated since it comes from known mirrors. This approach has minimal impact on the load of the servers and doesn't really require signing of metadata (which by itself does not solve the issue).

Quoting: But because of our "close servers" implementation, we will have more than 10 show up everywhere. So for some locations, a particular mirror might be in the list more often than in other locations (the US will have a different mirrorlist all the time, for example, while some other locations might get one or two on the list all the time).
That explains my observation.

Quoting: WRT only one mirror, if the user picks 2 or 3 instead of 1 and then uses the random (default) method for yum to pick mirrors, then they will still not need to worry about delayed updates. the only way the delayed updates is really a problem is if someone picks only one mirror. I would never do or recommend that.
Yes choosing all fake mirrors (or using only one) would be the case for CentOS/Fedora where the mentioned attack is possible. Debian based distros do not support mirror lists but have only one mirror so they are more vulnerable.

edit: My suggested approach has an issue that it assumes that all mirrors sync up quickly with the repomd mirrors. The package manager should try other mirrors too (if a package is not found) to handle this and it would be good behaviour to report to the user if a package is not found on any of the mirrors and ask him to create a better mirror list.

Posting in this forum is limited to members of the group: [ForumMods, SITEADMINS, MEMBERS.]

Becoming a member of LXer is easy and free. Join Us!