A couple of days ago I rescued the first farm right next to Sanctuary(I cannot remember the name to save my life, it's about a two minute walk after the Sanctuary bridge) from the raiders. They promised to join the Minutemen and I went on my merry way.

Yesterday or so, I recieved a notice that the settlement was under attack, but I turned off the marker in my pip boy because I like exploring.

Last night a notice appeared on my screen saying that my mission to protect the settlement had failed. Is this something I need to be concerned about?

I'm asking because aside from buying my house in Diamond Town and helping Preston build some turrents and crops I really have not done any settlement building.

I'm pretty sure you can completely ignore settlements and settlement building and still play the game without issue. That's the point to an open-ended world.

"Abernathy Farms and Ten Pines Bluffs are on fire."

Let 'em burn - The Vault Dweller/ Chosen one

Thanks for the answer. I'm not called the Lone Wanderer for nothing!

I've run into something similar in my game on a couple of occasions, so this is anecdotal evidence at best. The short answer is "not really". As best as I can tell, failing to defend a settlement under attack might cause the settlement's happiness to drop. But this assumes that there were even settlers there. The game has popped settlements onto my map just because I killed off whatever was there in the first place, not because I had created a settlement. It might also affect their willingness to spawn a Minuteman helper for a later mission, but I don't know the formula that determines when or if one spawns in the first place. Suffice to say that failing this early in the game doesn't seem to keep you from swimming in settlements in the late game, nor does it seem to impact your standing with the Minutemen.