Confusing Boyfriend, need advice on relationship!

My problem is this. I have been with this guy for 6 months now, he has always been the type to not want to get "serious". About a month ago he went on a trip for a week. All though before the trip he felt we were getting to serious for him :-/. Anyways, we decided to take a break for awhile, he wanted to go out with other girls and still date me...and i kinda felt the same way but wanted to just go out with other guys. We came to the agreement that we weren't out to make out with others and just see what would happen with all this. While he was gone a guy i knew kissed me, i didn;t kiss him back though..i missed my boyfriend and my boyfriend did not cheat on me or whatever while he was gone. I not being a virgin and he is, he claims he will not tell me he loves me until he is sure of it and wants to settle down and make it the right time. When he got back from his trip i cried and cried and told him i wanted to be with him and only him , and he still said he needed "time".. That night however, he lost his virginity to me.....yeah real "shocker" ever since things have changed better and bad. We spend more time together, but dont have sex that often, but when i ask him what hes doing (as in where he will be at times) he wont tell me, he just says he's going " out" i trust him very much are relationship is very free. Im wanting to know if i should say something to him about how i feel and why we cant be more serious? or just let things go and see what happens? please reply back.