what planet is the most radioactive?
So as I understand it the source of most geothermal energy on Earth is due to radioactive decay. Jupiter is the most massive planet but if the bulk of it's mass comes from gaseous elements instead of heavier ones like uranium or whatever than it being the most massive might not necessarily mean it has the most radioactive material.
There's a few parts to this question, I'll give it a shot. Also, since you're asking about the geothermal energy of the earth, I'm going to ignore radiation from space and radioactive van allen belts (Jupiter has a very radioactive van Allen belt). Also, as voyager discovered when it left the solar system, there's more radiation in the form of cosmic rays out there, that the sun effectively shields most of the planets from.
But, because you mention the Earth's geothermal heat from radiation, so I'll focus on that.
Here's a very well answered similar question about radioactive elements inside the earth. what percentage of the Earth's core is Uranium
So as I understand it the source of most geothermal energy on Earth is
due to radioactive decay.
A healthy share of Earth's core heat is from the planets formation. This is a pretty good article on that subject - http://physicsworld.com/cws/article/news/2011/jul/19/radioactive-decay-accounts-for-half-of-earths-heat, it says that of the current heat, about 50% is from the energy left from the formation of the planet and about 50% is from radiation - so the radiation is very important, but it's probably not the cause of "most" of the heat in the earth.
As far as I know, no similar studies have been done on any of the other planets.
Jupiter is the most massive planet but if the bulk of it's mass comes
from gaseous elements instead of heavier ones like uranium or whatever
than it being the most massive might not necessarily mean it has the
most radioactive material.
That I pretty much agree with. The two primary differences between the 4 gas giant planets (or, two gas giant, two ice giants if you like) and the 4 inner rocky planets is that the gas giants formed outside the frost line and key difference #2, the 4 outer planets are much larger, so they were more able to hold onto more hydrogen and helium, which are the 2 most abundant elements in the solar system.
Obviously, the 4 outer planets have a lower percentage of heavy elements overall. As to their percentage of Uranium - well, your guess is as good as mine, but, probably a lower percentage of radioactive elements than the inner planets seems reasonable to assume.
Also, any radioactive elements in the 4 outer planets would likely be well below their thick gaseous surfaces, so it would be difficult to detect, outside of Neutrino detection I suppose, and more trouble than it's worth to try to collect.
The 4 inner planets (and lets throw the moon in there as well, and asteroids if you like) are more interesting I'm assuming David Hammen's answer is correct in the linked question above, and that most of the Uranium on earth is in the crust or mantle, the highest percentage in the crust, and so that's probably true for any other rocky planet that formed and was, at one time, molten enough to circulate, so all the inner planets probably have similarly small amounts of uranium and other radioactive elements as Earth.
Mercury may have had much of it's crust and some of it's mantle blown off in a giant collision, and if that's the case, Mercury might have lost a portion of it's radioactive elements. Brief summary of that here, point #6: http://messenger.jhuapl.edu/faq/faq_science.html
unrelated footnote, but if Mercury's crust was blown off, it might be a good planet to mine for rare earth elements, or maybe asteroids will be easier, who knows, but I've always been curious of the mining possibilities on Mercury. Abundant solar energy too, but it's a hard planet to fly to because of the sun's gravity, so, maybe asteroids are the way to go for space mining.
The moon was also formed by a giant impact, so, the moon was formed by material from Earth's but it has a much higher percentage of mantle than the Earth, so it could have a higher percentage of Uranium and other radioactive elements than the earth. How well the Moon's material was circulated in it's cooling process is another question. Being smaller it probably cooled more quickly.
Venus and Mars, I see no reason why they shouldn't have similar percentages of radioactive elements than Earth.
One way to look at this question is to investigate how quickly the other inner planets cooled. Far from exact, but that probably tells us something, since heat generated by radioactive decay inside the planet would probably lead to volcanism at some point. Even the Moon appears to have had volcanic eruptions as recently as 33 million years ago, so some ongoing internal heating seems likely.
Size is also a key factor in what keeps the inside of a planet hot, and in some cases, tidal heating (Io, Enceladus), but that's not really relevant to this question.
Neither Mars nor Venus are volcanically active like the earth, but Olympus Mons may have erupted as recently as 25 million years ago, and there is evidence of large lava flows on Venus. As to how big a role radioactivity plays in heating up the cores of those 2 planets, I don't know precisely how much.
Overall, I think this is a good question on planet cores and radioactivity, as the inside of the planet is just as interesting as the surface, well, to me, at least. I know this doesn't answer your question specifically as to which is the most radioactive, cause I don't know, but I like the question.
Are objects with higher density expected to be more radioactive, or only objects which are (or once were) large? For example, asteroid 16 Psyche has a density of 6.7 (between 4 and 10 anyway) but has only 186 km mean diameter. Should it be more or less radioactive than Earth?
I don't know the answer to that one. Pure uranium is more dense than most minerals but it's also fairly chemically reactive where as, something like Iridium is non reactive and it tends to stay pure Iridium, which is why it's more common on meteors than the Earth's surface. I think because of the way planets mixed as they formed, Uranium would probobly be more common on planet crusts than in asteroids, but that's kind of a guess. There could be other factors with asteroids. I'm not sure.
Small footnote, but I don't really like the term "more radioactive" in this context. it's not a good way to describe a planet or asteroid, which is probobly why this question got some close votes. An asteroid might have Uranium, but if it's mixed throughout and the heat from decay is able to conduct to the surface and then radiate away, the surface of the asteroid wouldn't necessarily be radioactive at all, even with higher Uranium content. Better to ask in parts per billion concentrations, etc. - just wanted to add that.
There's actually some indirect evidence for ongoing volcanic activity on Venus: http://www.esa.int/Our_Activities/Space_Science/Venus_Express/Hot_lava_flows_discovered_on_Venus