Usability matters, speed doesn't
Over time computers have increased in speed, programming languages can do much more with fewer lines of code and memory is not a problem like it used to be. What this means is that unless your site is large, you don't really need to concern yourself with speed. You don't have to continually try to improve your code for speed increases.
All websites need to be easy to use. Forms and navigation need to be properly constructed. Text size and colour is important. Accessibility should be thought about. Above all else make your website easy to use.
If you are extremely lucky, your website may become large. Once it starts to get bigger, you will notice a lag in the speed of which pages are downloaded. At this point do some refactoring and concentrate on getting the website running smoothly.
But for most websites this will never eventuate. You will never need to refactor. What you will need to do is ensure that the frontend of the website has been built correctly. Concentrate on this, it's they most important aspect of your website.
99% of sites need to be easier to use. 1% of sites that are easy to use, need to be faster.
