Yes and no. It's about size in that once an object gets to a certain size, it becomes spherical due to its own gravity, but the third determining factor (which is where things like Pluto and Eris fail) is all about location.
The three things that are used to determine whether or not an object is a planet (according to the IAU) are:
1. It orbits the Sun.
2. It has sufficient mass to achieve hydrostatic equilibrium (meaning that it is big enough that its gravity makes it roughly spherical)
3. It has "cleared its neighborhood."
That third point means that a body has become the dominant gravitational influence in its orbital zone, and there are no other large bodies inside the orbital zone that aren't its own satellites or similarly under its gravitational influence.
That's where location comes into play. Basically, the further a planet is away from the sun, the harder it becomes for it to clear its neighborhood. If you put Pluto where Mercury is, it would be a planet. But because of its location, it isn't considered one.
It becomes truly ridiculous when you consider the fact that half the planets still called planets in our solar system, Earth, Mars, Jupiter, and Neptune, haven't cleared their neighborhoods. Jupiter's orbital zone is home to more than a million asteroids. If Neptune had cleared its neighborhood, Pluto wouldn't be where it is.
The clearing its neighborhood argument is like saying your cat is a cat as long as it isn't on a chair. If it's on a chair, it must be considered a mongoose.